Date: (Mon) May 30, 2016

Introduction:

Data: Source: Training: “https://inclass.kaggle.com/c/can-we-predict-voting-outcomes/download/train2016.csv”
New: “https://inclass.kaggle.com/c/can-we-predict-voting-outcomes/download/test2016.csv”
Time period:

Synopsis:

Based on analysis utilizing <> techniques, :

Summary of key steps & error improvement stats:

Prediction Accuracy Enhancement Options:

transform.data chunk:
- derive features from multiple features
manage.missing.data chunk:
- Not fill missing vars
- Fill missing numerics with a different algorithm
- Fill missing chars with data based on clusters

[](.png)

Potential next steps include:

Organization:
- Categorize by chunk
- Priority criteria:
  1. Ease of change
  2. Impacts report
  3. Cleans innards
  4. Bug report
all chunks:
- at chunk-end rm(!glb_)
manage.missing.data chunk:
- cleaner way to manage re-splitting of training vs. new entity
extract.features chunk:
- Add n-grams for glbFeatsText
  - “RTextTools”, “tau”, “RWeka”, and “textcat” packages
fit.models chunk:
- Classification: Plot AUC Curves for all models & highlight glbMdlSel
- Prediction accuracy scatter graph:
- Add tiles (raw vs. PCA)
- Use shiny for drop-down of “important” features
- Use plot.ly for interactive plots ?
- Change .fit suffix of model metrics to .mdl if it’s data independent (e.g. AIC, Adj.R.Squared - is it truly data independent ?, etc.)
- create a custom model for rpart that has minbucket as a tuning parameter
- varImp for randomForest crashes in caret version:6.0.41 -> submit bug report
Probability handling for multinomials vs. desired binomial outcome
ROCR currently supports only evaluation of binary classification tasks (version 1.0.7)
extensions toward multiclass classification are scheduled for the next release
fit.all.training chunk:
- myplot_prediction_classification: displays ‘x’ instead of ‘+’ when there are no prediction errors
Compare glb_sel_mdl vs. glb_fin_mdl:
- varImp
- Prediction differences (shd be minimal ?)
Move glb_analytics_diag_plots to mydsutils.R: (+) Easier to debug (-) Too many glb vars used
Add print(ggplot.petrinet(glb_analytics_pn) + coord_flip()) at the end of every major chunk
Parameterize glb_analytics_pn
Move glb_impute_missing_data to mydsutils.R: (-) Too many glb vars used; glb_<>_df reassigned
Do non-glm methods handle interaction terms ?
f-score computation for classifiers should be summation across outcomes (not just the desired one ?)
Add accuracy computation to glb_dmy_mdl in predict.data.new chunk
Why does splitting fit.data.training.all chunk into separate chunks add an overhead of ~30 secs ? It’s not rbind b/c other chunks have lower elapsed time. Is it the number of plots ?
Incorporate code chunks in print_sessionInfo
Test against
- projects in github.com/bdanalytics
- lectures in jhu-datascience track

Analysis:

rm(list = ls())
set.seed(12345)
options(stringsAsFactors = FALSE)
source("~/Dropbox/datascience/R/mycaret.R")
source("~/Dropbox/datascience/R/mydsutils.R")

## Loading required package: caret

## Loading required package: lattice

## Loading required package: ggplot2

source("~/Dropbox/datascience/R/mypetrinet.R")
source("~/Dropbox/datascience/R/myplclust.R")
source("~/Dropbox/datascience/R/myplot.R")
source("~/Dropbox/datascience/R/myscript.R")
source("~/Dropbox/datascience/R/mytm.R")
# Gather all package requirements here
suppressPackageStartupMessages(require(doMC))
glbCores <- 6 # of cores on machine - 2
registerDoMC(glbCores) 

suppressPackageStartupMessages(require(caret))
require(plyr)

## Loading required package: plyr

require(dplyr)

## Loading required package: dplyr

## 
## Attaching package: 'dplyr'

## The following objects are masked from 'package:plyr':
## 
##     arrange, count, desc, failwith, id, mutate, rename, summarise,
##     summarize

## The following objects are masked from 'package:stats':
## 
##     filter, lag

## The following objects are masked from 'package:base':
## 
##     intersect, setdiff, setequal, union

require(knitr)

## Loading required package: knitr

require(stringr)

## Loading required package: stringr

#source("dbgcaret.R")
#packageVersion("snow")
#require(sos); findFn("cosine", maxPages=2, sortby="MaxScore")

# Analysis control global variables
# Inputs
#   url/name = "<PathPointer>"; if url specifies a zip file, name = "<filename>"; 
#               or named collection of <PathPointer>s
#   sep = choose from c(NULL, "\t")
glbObsTrnFile <- list(url = "https://inclass.kaggle.com/c/can-we-predict-voting-outcomes/download/train2016.csv"
    # or list(url = c(NULL, <.inp1> = "<path1>", <.inp2> = "<path2>"))
    #, splitSpecs = list(method = "copy" # default when glbObsNewFile is NULL
    #                       select from c("copy", NULL ???, "condition", "sample", )
    #                      ,nRatio = 0.3 # > 0 && < 1 if method == "sample" 
    #                      ,seed = 123 # any integer or glbObsTrnPartitionSeed if method == "sample" 
    #                      ,condition = # or 'is.na(<var>)'; '<var> <condition_operator> <value>'    
    #                      )
    )                   
 
glbObsNewFile <- list(url = "https://inclass.kaggle.com/c/can-we-predict-voting-outcomes/download/test2016.csv") 

glbObsDropCondition <- NULL # : default
#   enclose in single-quotes b/c condition might include double qoutes
#       use | & ; NOT || &&    
#   '<condition>' 
    # 'grepl("^First Draft Video:", glbObsAll$Headline)'
    # 'is.na(glbObsAll[, glb_rsp_var_raw])'
    # '(is.na(glbObsAll[, glb_rsp_var_raw]) & grepl("Train", glbObsAll[, glbFeatsId]))'
    # 'is.na(strptime(glbObsAll[, "Date"], glbFeatsDateTime[["Date"]]["format"], tz = glbFeatsDateTime[["Date"]]["timezone"]))'
#nrow(do.call("subset",list(glbObsAll, parse(text=paste0("!(", glbObsDropCondition, ")")))))
    
glb_obs_repartition_train_condition <- NULL # : default
#    "<condition>" 

glb_max_fitobs <- NULL # or any integer
glbObsTrnPartitionSeed <- 123 # or any integer
                         
glb_is_regression <- FALSE; glb_is_classification <- !glb_is_regression; 
    glb_is_binomial <- TRUE # or TRUE or FALSE

glb_rsp_var_raw <- "Party"

# for classification, the response variable has to be a factor
glb_rsp_var <- "Party.fctr"

# if the response factor is based on numbers/logicals e.g (0/1 OR TRUE/FALSE vs. "A"/"B"), 
#   or contains spaces (e.g. "Not in Labor Force")
#   caret predict(..., type="prob") crashes
glb_map_rsp_raw_to_var <- #NULL 
function(raw) {
#     return(raw ^ 0.5)
#     return(log(raw))
#     return(log(1 + raw))
#     return(log10(raw)) 
#     return(exp(-raw / 2))
    ret_vals <- rep_len(NA, length(raw)); ret_vals[!is.na(raw)] <- ifelse(raw[!is.na(raw)] == "Republican", "R", "D"); return(relevel(as.factor(ret_vals), ref = "R"))
#     as.factor(paste0("B", raw))
#     as.factor(gsub(" ", "\\.", raw))
    }

#if glb_rsp_var_raw is numeric:
#print(summary(glbObsAll[, glb_rsp_var_raw]))
#glb_map_rsp_raw_to_var(tst <- c(NA, as.numeric(summary(glbObsAll[, glb_rsp_var_raw])))) 

#if glb_rsp_var_raw is character:
#print(table(glbObsAll[, glb_rsp_var_raw], useNA = "ifany"))
# print(table(glb_map_rsp_raw_to_var(tst <- glbObsAll[, glb_rsp_var_raw]), useNA = "ifany"))

glb_map_rsp_var_to_raw <- #NULL 
function(var) {
#     return(var ^ 2.0)
#     return(exp(var))
#     return(10 ^ var) 
#     return(-log(var) * 2)
#     as.numeric(var)
#     levels(var)[as.numeric(var)]
    sapply(levels(var)[as.numeric(var)], function(elm) 
        if (is.na(elm)) return(elm) else
        if (elm == 'R') return("Republican") else
        if (elm == 'D') return("Democrat") else
        stop("glb_map_rsp_var_to_raw: unexpected value: ", elm)
        )  
#     gsub("\\.", " ", levels(var)[as.numeric(var)])
#     c("<=50K", " >50K")[as.numeric(var)]
#     c(FALSE, TRUE)[as.numeric(var)]
}
# print(table(glb_map_rsp_var_to_raw(glb_map_rsp_raw_to_var(tst)), useNA = "ifany"))

if ((glb_rsp_var != glb_rsp_var_raw) && is.null(glb_map_rsp_raw_to_var))
    stop("glb_map_rsp_raw_to_var function expected")

# List info gathered for various columns
# <col_name>:   <description>; <notes>
# USER_ID - an anonymous id unique to a given user
# YOB - the year of birth of the user
# Gender - the gender of the user, either Male or Female
# Income - the household income of the user. Either not provided, or one of "under $25,000", "$25,001 - $50,000", "$50,000 - $74,999", "$75,000 - $100,000", "$100,001 - $150,000", or "over $150,000".
# HouseholdStatus - the household status of the user. Either not provided, or one of "Domestic Partners (no kids)", "Domestic Partners (w/kids)", "Married (no kids)", "Married (w/kids)", "Single (no kids)", or "Single (w/kids)".
# EducationalLevel - the education level of the user. Either not provided, or one of "Current K-12", "High School Diploma", "Current Undergraduate", "Associate's Degree", "Bachelor's Degree", "Master's Degree", or "Doctoral Degree".
# Party - the political party for whom the user intends to vote for. Either "Democrat" or "Republican
# Q124742, Q124122, . . . , Q96024 - 101 different questions that the users were asked on Show of Hands. If the user didn't answer the question, there is a blank. For information about the question text and possible answers, see the file Questions.pdf.

# currently does not handle more than 1 column; consider concatenating multiple columns
# If glbFeatsId == NULL, ".rownames <- as.numeric(row.names())" is the default
glbFeatsId <- "USER_ID" # choose from c(NULL : default, "<id_feat>") 
glbFeatsCategory <- "Hhold.fctr" # choose from c(NULL : default, "<category_feat>")
# glbFeatsCategory <- "Q109244.fctr" # choose from c(NULL : default, "<category_feat>") -> OOB performed worse than "Hhold.fctr"

# User-specified exclusions
glbFeatsExclude <- c(NULL
#   Feats that shd be excluded due to known causation by prediction variable
# , "<feat1", "<feat2>"
#   Feats that are factors with unique values (as % of nObs) > 49 (empirically derived)
#   Feats that are linear combinations (alias in glm)
#   Feature-engineering phase -> start by excluding all features except id & category & 
#       work each one in
    , "USER_ID", "YOB", "Gender", "Income", "HouseholdStatus", "EducationLevel" 
    ,"Q124742","Q124122" 
    ,"Q123621","Q123464"
    ,"Q122771","Q122770","Q122769","Q122120"
    ,"Q121700","Q121699","Q121011"
    ,"Q120978","Q120650","Q120472","Q120379","Q120194","Q120014","Q120012" 
    ,"Q119851","Q119650","Q119334"
    ,"Q118892","Q118237","Q118233","Q118232","Q118117"
    ,"Q117193","Q117186"
    ,"Q116797","Q116881","Q116953","Q116601","Q116441","Q116448","Q116197"
    ,"Q115602","Q115777","Q115610","Q115611","Q115899","Q115390","Q115195"
    ,"Q114961","Q114748","Q114517","Q114386","Q114152"
    ,"Q113992","Q113583","Q113584","Q113181"
    ,"Q112478","Q112512","Q112270"
    ,"Q111848","Q111580","Q111220"
    ,"Q110740"
    ,"Q109367","Q109244"
    ,"Q108950","Q108855","Q108617","Q108856","Q108754","Q108342","Q108343"
    ,"Q107869","Q107491"
    ,"Q106993","Q106997","Q106272","Q106388","Q106389","Q106042"
    ,"Q105840","Q105655"
    ,"Q104996"
    ,"Q103293"
    ,"Q102906","Q102674","Q102687","Q102289","Q102089"
    ,"Q101162","Q101163","Q101596"
    ,"Q100689","Q100680","Q100562","Q100010"
    ,"Q99982"
    ,"Q99716"
    ,"Q99581"
    ,"Q99480"
    ,"Q98869"
    ,"Q98578"
    ,"Q98197"
    ,"Q98059","Q98078"
    ,"Q96024" # Done
    ,".pos") 
if (glb_rsp_var_raw != glb_rsp_var)
    glbFeatsExclude <- union(glbFeatsExclude, glb_rsp_var_raw)                    

glbFeatsInteractionOnly <- list()
#glbFeatsInteractionOnly[["<child_feat>"]] <- "<parent_feat>"

glbFeatsDrop <- c(NULL
                # , "<feat1>", "<feat2>"
                )

glb_map_vars <- NULL # or c("<var1>", "<var2>")
glb_map_urls <- list();
# glb_map_urls[["<var1>"]] <- "<var1.url>"

# Derived features; Use this mechanism to cleanse data ??? Cons: Data duplication ???
glbFeatsDerive <- list();

# glbFeatsDerive[["<feat.my.sfx>"]] <- list(
#     mapfn = function(<arg1>, <arg2>) { return(function(<arg1>, <arg2>)) } 
#   , args = c("<arg1>", "<arg2>"))
#myprint_df(data.frame(ImageId = mapfn(glbObsAll$.src, glbObsAll$.pos)))
#data.frame(ImageId = mapfn(glbObsAll$.src, glbObsAll$.pos))[7045:7055, ]

    # character
#     mapfn = function(Education) { raw <- Education; raw[is.na(raw)] <- "NA.my"; return(as.factor(raw)) } 
#     mapfn = function(Week) { return(substr(Week, 1, 10)) }
#     mapfn = function(Name) { return(sapply(Name, function(thsName) 
#                                             str_sub(unlist(str_split(thsName, ","))[1], 1, 1))) } 

#     mapfn = function(descriptor) { return(plyr::revalue(descriptor, c(
#         "ABANDONED BUILDING"  = "OTHER",
#         "**"                  = "**"
#                                           ))) }

#     mapfn = function(description) { mod_raw <- description;
    # This is here because it does not work if it's in txt_map_filename
#         mod_raw <- gsub(paste0(c("\n", "\211", "\235", "\317", "\333"), collapse = "|"), " ", mod_raw)
    # Don't parse for "." because of ".com"; use customized gsub for that text
#         mod_raw <- gsub("(\\w)(!|\\*|,|-|/)(\\w)", "\\1\\2 \\3", mod_raw);
    # Some state acrnoyms need context for separation e.g. 
    #   LA/L.A. could either be "Louisiana" or "LosAngeles"
        # modRaw <- gsub("\\bL\\.A\\.( |,|')", "LosAngeles\\1", modRaw);
    #   OK/O.K. could either be "Oklahoma" or "Okay"
#         modRaw <- gsub("\\bACA OK\\b", "ACA OKay", modRaw); 
#         modRaw <- gsub("\\bNow O\\.K\\.\\b", "Now OKay", modRaw);        
    #   PR/P.R. could either be "PuertoRico" or "Public Relations"        
        # modRaw <- gsub("\\bP\\.R\\. Campaign", "PublicRelations Campaign", modRaw);        
    #   VA/V.A. could either be "Virginia" or "VeteransAdministration"        
        # modRaw <- gsub("\\bthe V\\.A\\.\\:", "the VeteranAffairs:", modRaw);
    #   
    # Custom mods

#         return(mod_raw) }

    # numeric
# Create feature based on record position/id in data   
glbFeatsDerive[[".pos"]] <- list(
    mapfn = function(raw1) { return(1:length(raw1)) }
    , args = c(".rnorm"))
# glbFeatsDerive[[".pos.y"]] <- list(
#     mapfn = function(raw1) { return(1:length(raw1)) }       
#     , args = c(".rnorm"))    

# Add logs of numerics that are not distributed normally
#   Derive & keep multiple transformations of the same feature, if normality is hard to achieve with just one transformation
#   Right skew: logp1; sqrt; ^ 1/3; logp1(logp1); log10; exp(-<feat>/constant)
# glbFeatsDerive[["WordCount.log1p"]] <- list(
#     mapfn = function(WordCount) { return(log1p(WordCount)) } 
#   , args = c("WordCount"))
# glbFeatsDerive[["WordCount.root2"]] <- list(
#     mapfn = function(WordCount) { return(WordCount ^ (1/2)) } 
#   , args = c("WordCount"))
# glbFeatsDerive[["WordCount.nexp"]] <- list(
#     mapfn = function(WordCount) { return(exp(-WordCount)) } 
#   , args = c("WordCount"))
#print(summary(glbObsAll$WordCount))
#print(summary(mapfn(glbObsAll$WordCount)))
    
# If imputation shd be skipped for this feature
# glbFeatsDerive[["District.fctr"]] <- list(
#     mapfn = function(District) {
#         raw <- District;
#         ret_vals <- rep_len("NA", length(raw)); 
#         ret_vals[!is.na(raw)] <- sapply(raw[!is.na(raw)], function(elm) 
#                                         ifelse(elm < 10, "1-9", 
#                                         ifelse(elm < 20, "10-19", "20+")));
#         return(relevel(as.factor(ret_vals), ref = "NA"))
#     }       
#     , args = c("District"))    

# YOB options:
# 1. Missing data:
# 1.1   0 -> Does not improve baseline
# 1.2   Cut factors & "NA" is a level
# 2. Data corrections: < 1928 & > 2000
# 3. Scale YOB
# 4. Add Age
glbFeatsDerive[["YOB.Age.fctr"]] <- list(
    mapfn = function(raw1) {
        raw <- 2016 - raw1 
        # raw[!is.na(raw) & raw >= 2010] <- NA
        raw[!is.na(raw) & (raw <= 15)] <- NA
        raw[!is.na(raw) & (raw >= 90)] <- NA        
        retVal <- rep_len("NA", length(raw))
        # breaks = c(1879, seq(1949, 1989, 10), 2049)
        # cutVal <- cut(raw[!is.na(raw)], breaks = breaks, 
        #               labels = as.character(breaks + 1)[1:(length(breaks) - 1)])
        cutVal <- cut(raw[!is.na(raw)], breaks = c(15, 20, 25, 30, 35, 40, 50, 65, 90))
        retVal[!is.na(raw)] <- levels(cutVal)[cutVal]
        return(factor(retVal, levels = c("NA"
                ,"(15,20]","(20,25]","(25,30]","(30,35]","(35,40]","(40,50]","(50,65]","(65,90]"),
                        ordered = TRUE))
    }
    , args = c("YOB"))

glbFeatsDerive[["Gender.fctr"]] <- list(
    mapfn = function(raw1) {
        raw <- raw1
        raw[raw %in% ""] <- "N"
        raw <- gsub("Male"  , "M", raw, fixed = TRUE)
        raw <- gsub("Female", "F", raw, fixed = TRUE)        
        return(relevel(as.factor(raw), ref = "N"))
    }
    , args = c("Gender"))

glbFeatsDerive[["Income.fctr"]] <- list(
    mapfn = function(raw1) { raw <- raw1;
        raw[raw %in% ""] <- "N"
        raw <- gsub("under $25,000"      , "<25K"    , raw, fixed = TRUE)
        raw <- gsub("$25,001 - $50,000"  , "25-50K"  , raw, fixed = TRUE)
        raw <- gsub("$50,000 - $74,999"  , "50-75K"  , raw, fixed = TRUE)
        raw <- gsub("$75,000 - $100,000" , "75-100K" , raw, fixed = TRUE)        
        raw <- gsub("$100,001 - $150,000", "100-150K", raw, fixed = TRUE)
        raw <- gsub("over $150,000"      , ">150K"   , raw, fixed = TRUE)        
        return(factor(raw, levels = c("N","<25K","25-50K","50-75K","75-100K","100-150K",">150K"),
                      ordered = TRUE))
    }
    , args = c("Income"))

glbFeatsDerive[["Hhold.fctr"]] <- list(
    mapfn = function(raw1) { raw <- raw1;
        raw[raw %in% ""] <- "N"
        raw <- gsub("Domestic Partners (no kids)", "PKn", raw, fixed = TRUE)
        raw <- gsub("Domestic Partners (w/kids)" , "PKy", raw, fixed = TRUE)        
        raw <- gsub("Married (no kids)"          , "MKn", raw, fixed = TRUE)
        raw <- gsub("Married (w/kids)"           , "MKy", raw, fixed = TRUE)        
        raw <- gsub("Single (no kids)"           , "SKn", raw, fixed = TRUE)
        raw <- gsub("Single (w/kids)"            , "SKy", raw, fixed = TRUE)        
        return(relevel(as.factor(raw), ref = "N"))
    }
    , args = c("HouseholdStatus"))

glbFeatsDerive[["Edn.fctr"]] <- list(
    mapfn = function(raw1) { raw <- raw1;
        raw[raw %in% ""] <- "N"
        raw <- gsub("Current K-12"         , "K12", raw, fixed = TRUE)
        raw <- gsub("High School Diploma"  , "HSD", raw, fixed = TRUE)        
        raw <- gsub("Current Undergraduate", "CCg", raw, fixed = TRUE)
        raw <- gsub("Associate's Degree"   , "Ast", raw, fixed = TRUE)
        raw <- gsub("Bachelor's Degree"    , "Bcr", raw, fixed = TRUE)        
        raw <- gsub("Master's Degree"      , "Msr", raw, fixed = TRUE)
        raw <- gsub("Doctoral Degree"      , "PhD", raw, fixed = TRUE)        
        return(factor(raw, levels = c("N","K12","HSD","CCg","Ast","Bcr","Msr","PhD"),
                      ordered = TRUE))
    }
    , args = c("EducationLevel"))

# for (qsn in c("Q124742","Q124122"))
# for (qsn in grep("Q12(.{4})(?!\\.fctr)", names(glbObsTrn), value = TRUE, perl = TRUE))
for (qsn in grep("Q", glbFeatsExclude, fixed = TRUE, value = TRUE))    
    glbFeatsDerive[[paste0(qsn, ".fctr")]] <- list(
        mapfn = function(raw1) {
            raw1[raw1 %in% ""] <- "NA"
            rawVal <- unique(raw1)
            
            if (length(setdiff(rawVal, (expVal <- c("NA", "No", "Ys")))) == 0) {
                raw1 <- gsub("Yes", "Ys", raw1, fixed = TRUE)
                if (length(setdiff(rawVal, expVal)) > 0)
                    stop(qsn, " vals: ", paste0(rawVal, collapse = "|"), 
                         " does not match expectation: ", paste0(expVal, collapse = "|"))
            } else
            if (length(setdiff(rawVal, (expVal <- c("NA", "Me", "Circumstances")))) == 0) {
                raw1 <- gsub("Circumstances", "Cs", raw1, fixed = TRUE)
                if (length(setdiff(rawVal, expVal)) > 0)
                    stop(qsn, " vals: ", paste0(rawVal, collapse = "|"), 
                         " does not match expectation: ", paste0(expVal, collapse = "|"))
            } else
            if (length(setdiff(rawVal, (expVal <- c("NA", "Grrr people", "Yay people!")))) == 0) {
                raw1 <- gsub("Grrr people", "Gr", raw1, fixed = TRUE)
                raw1 <- gsub("Yay people!", "Yy", raw1, fixed = TRUE)                
                if (length(setdiff(rawVal, expVal)) > 0)
                    stop(qsn, " vals: ", paste0(rawVal, collapse = "|"), 
                         " does not match expectation: ", paste0(expVal, collapse = "|"))
            } else
            if (length(setdiff(rawVal, (expVal <- c("NA", "Idealist", "Pragmatist")))) == 0) {
                raw1 <- gsub("Idealist"  , "Id", raw1, fixed = TRUE)
                raw1 <- gsub("Pragmatist", "Pr", raw1, fixed = TRUE)                
                if (length(setdiff(rawVal, expVal)) > 0)
                    stop(qsn, " vals: ", paste0(rawVal, collapse = "|"), 
                         " does not match expectation: ", paste0(expVal, collapse = "|"))
            } else
            if (length(setdiff(rawVal, (expVal <- c("NA", "Private", "Public")))) == 0) {
                raw1 <- gsub("Private", "Pt", raw1, fixed = TRUE)
                raw1 <- gsub("Public" , "Pc", raw1, fixed = TRUE)                
                if (length(setdiff(rawVal, expVal)) > 0)
                    stop(qsn, " vals: ", paste0(rawVal, collapse = "|"), 
                         " does not match expectation: ", paste0(expVal, collapse = "|"))
            }
            
            return(relevel(as.factor(raw1), ref = "NA"))
        }
        , args = c(qsn))

# If imputation of missing data is not working ...
# glbFeatsDerive[["FertilityRate.nonNA"]] <- list(
#     mapfn = function(FertilityRate, Region) {
#         RegionMdn <- tapply(FertilityRate, Region, FUN = median, na.rm = TRUE)
# 
#         retVal <- FertilityRate
#         retVal[is.na(FertilityRate)] <- RegionMdn[Region[is.na(FertilityRate)]]
#         return(retVal)
#     }
#     , args = c("FertilityRate", "Region"))
    
#     mapfn = function(HOSPI.COST) { return(cut(HOSPI.COST, 5, breaks = c(0, 100000, 200000, 300000, 900000), labels = NULL)) }     
#     mapfn = function(Rasmussen)  { return(ifelse(sign(Rasmussen) >= 0, 1, 0)) } 
#     mapfn = function(startprice) { return(startprice ^ (1/2)) }       
#     mapfn = function(startprice) { return(log(startprice)) }   
#     mapfn = function(startprice) { return(exp(-startprice / 20)) }
#     mapfn = function(startprice) { return(scale(log(startprice))) }     
#     mapfn = function(startprice) { return(sign(sprice.predict.diff) * (abs(sprice.predict.diff) ^ (1/10))) }        

    # factor      
#     mapfn = function(PropR) { return(as.factor(ifelse(PropR >= 0.5, "Y", "N"))) }
#     mapfn = function(productline, description) { as.factor(gsub(" ", "", productline)) }
#     mapfn = function(purpose) { return(relevel(as.factor(purpose), ref="all_other")) }
#     mapfn = function(raw) { tfr_raw <- as.character(cut(raw, 5)); 
#                             tfr_raw[is.na(tfr_raw)] <- "NA.my";
#                             return(as.factor(tfr_raw)) }
#     mapfn = function(startprice.log10) { return(cut(startprice.log10, 3)) }
#     mapfn = function(startprice.log10) { return(cut(sprice.predict.diff, c(-1000, -100, -10, -1, 0, 1, 10, 100, 1000))) }    

#     , args = c("<arg1>"))
    
    # multiple args
#     mapfn = function(id, date) { return(paste(as.character(id), as.character(date), sep = "#")) }        
#     mapfn = function(PTS, oppPTS) { return(PTS - oppPTS) }
#     mapfn = function(startprice.log10.predict, startprice) {
#                  return(spdiff <- (10 ^ startprice.log10.predict) - startprice) } 
#     mapfn = function(productline, description) { as.factor(
#         paste(gsub(" ", "", productline), as.numeric(nchar(description) > 0), sep = "*")) }
#     mapfn = function(.src, .pos) { 
#         return(paste(.src, sprintf("%04d", 
#                                    ifelse(.src == "Train", .pos, .pos - 7049)
#                                    ), sep = "#")) }       

# # If glbObsAll is not sorted in the desired manner
#     mapfn=function(Week) { return(coredata(lag(zoo(orderBy(~Week, glbObsAll)$ILI), -2, na.pad=TRUE))) }
#     mapfn=function(ILI) { return(coredata(lag(zoo(ILI), -2, na.pad=TRUE))) }
#     mapfn=function(ILI.2.lag) { return(log(ILI.2.lag)) }

# glbFeatsDerive[["<var1>"]] <- glbFeatsDerive[["<var2>"]]

# tst <- "descr.my"; args_lst <- NULL; for (arg in glbFeatsDerive[[tst]]$args) args_lst[[arg]] <- glbObsAll[, arg]; print(head(args_lst[[arg]])); print(head(drv_vals <- do.call(glbFeatsDerive[[tst]]$mapfn, args_lst))); 
# print(which_ix <- which(args_lst[[arg]] == 0.75)); print(drv_vals[which_ix]); 

glbFeatsDateTime <- list()
# Use OlsonNames() to enumerate supported time zones
# glbFeatsDateTime[["<DateTimeFeat>"]] <- 
#     c(format = "%Y-%m-%d %H:%M:%S" or "%m/%e/%y", timezone = "US/Eastern", impute.na = TRUE, 
#       last.ctg = FALSE, poly.ctg = FALSE)

glbFeatsPrice <- NULL # or c("<price_var>")

glbFeatsImage <- list() #list(<imageFeat> = list(patchSize = 10)) # if patchSize not specified, no patch computation

glbFeatsText <- list()
Sys.setlocale("LC_ALL", "C") # For english

## [1] "C/C/C/C/C/en_US.UTF-8"

#glbFeatsText[["<TextFeature>"]] <- list(NULL,
#   ,names = myreplacePunctuation(str_to_lower(gsub(" ", "", c(NULL, 
#       <comma-separated-screened-names>
#   ))))
#   ,rareWords = myreplacePunctuation(str_to_lower(gsub(" ", "", c(NULL, 
#       <comma-separated-nonSCOWL-words>
#   ))))
#)

# Text Processing Step: custom modifications not present in txt_munge -> use glbFeatsDerive
# Text Processing Step: universal modifications
glb_txt_munge_filenames_pfx <- "<projectId>_mytxt_"

# Text Processing Step: tolower
# Text Processing Step: myreplacePunctuation
# Text Processing Step: removeWords
glb_txt_stop_words <- list()
# Remember to use unstemmed words
if (length(glbFeatsText) > 0) {
    require(tm)
    require(stringr)

    glb_txt_stop_words[["<txt_var>"]] <- sort(myreplacePunctuation(str_to_lower(gsub(" ", "", c(NULL
        # Remove any words from stopwords            
#         , setdiff(myreplacePunctuation(stopwords("english")), c("<keep_wrd1>", <keep_wrd2>"))
                                
        # Remove salutations
        ,"mr","mrs","dr","Rev"                                

        # Remove misc
        #,"th" # Happy [[:digit::]]+th birthday 

        # Remove terms present in Trn only or New only; search for "Partition post-stem"
        #   ,<comma-separated-terms>        

        # cor.y.train == NA
#         ,unlist(strsplit(paste(c(NULL
#           ,"<comma-separated-terms>"
#         ), collapse=",")

        # freq == 1; keep c("<comma-separated-terms-to-keep>")
            # ,<comma-separated-terms>

        # chisq.pval high (e.g. == 1); keep c("<comma-separated-terms-to-keep>")
            # ,<comma-separated-terms>

        # nzv.freqRatio high (e.g. >= glbFeatsNzvFreqMax); keep c("<comma-separated-terms-to-keep>")
            # ,<comma-separated-terms>        
                                            )))))
}
#orderBy(~term, glb_post_stem_words_terms_df_lst[[txtFeat]][grep("^man", glb_post_stem_words_terms_df_lst[[txtFeat]]$term), ])
#glbObsAll[glb_post_stem_words_terms_mtrx_lst[[txtFeat]][, 4866] > 0, c(glb_rsp_var, txtFeat)]

# To identify terms with a specific freq
#paste0(sort(subset(glb_post_stop_words_terms_df_lst[[txtFeat]], freq == 1)$term), collapse = ",")
#paste0(sort(subset(glb_post_stem_words_terms_df_lst[[txtFeat]], freq <= 2)$term), collapse = ",")
#subset(glb_post_stem_words_terms_df_lst[[txtFeat]], term %in% c("zinger"))

# To identify terms with a specific freq & 
#   are not stemmed together later OR is value of color.fctr (e.g. gold)
#paste0(sort(subset(glb_post_stop_words_terms_df_lst[[txtFeat]], (freq == 1) & !(term %in% c("blacked","blemish","blocked","blocks","buying","cables","careful","carefully","changed","changing","chargers","cleanly","cleared","connect","connects","connected","contains","cosmetics","default","defaulting","defective","definitely","describe","described","devices","displays","drop","drops","engravement","excellant","excellently","feels","fix","flawlessly","frame","framing","gentle","gold","guarantee","guarantees","handled","handling","having","install","iphone","iphones","keeped","keeps","known","lights","line","lining","liquid","liquidation","looking","lots","manuals","manufacture","minis","most","mostly","network","networks","noted","opening","operated","performance","performs","person","personalized","photograph","physically","placed","places","powering","pre","previously","products","protection","purchasing","returned","rotate","rotation","running","sales","second","seconds","shipped","shuts","sides","skin","skinned","sticker","storing","thats","theres","touching","unusable","update","updates","upgrade","weeks","wrapped","verified","verify") ))$term), collapse = ",")

#print(subset(glb_post_stem_words_terms_df_lst[[txtFeat]], (freq <= 2)))
#glbObsAll[which(terms_mtrx[, 229] > 0), glbFeatsText]

# To identify terms with cor.y == NA
#orderBy(~-freq+term, subset(glb_post_stop_words_terms_df_lst[[txtFeat]], is.na(cor.y)))
#paste(sort(subset(glb_post_stop_words_terms_df_lst[[txtFeat]], is.na(cor.y))[, "term"]), collapse=",")
#orderBy(~-freq+term, subset(glb_post_stem_words_terms_df_lst[[txtFeat]], is.na(cor.y)))

# To identify terms with low cor.y.abs
#head(orderBy(~cor.y.abs+freq+term, subset(glb_post_stem_words_terms_df_lst[[txtFeat]], !is.na(cor.y))), 5)

# To identify terms with high chisq.pval
#subset(glb_post_stem_words_terms_df_lst[[txtFeat]], chisq.pval > 0.99)
#paste0(sort(subset(glb_post_stem_words_terms_df_lst[[txtFeat]], (chisq.pval > 0.99) & (freq <= 10))$term), collapse=",")
#paste0(sort(subset(glb_post_stem_words_terms_df_lst[[txtFeat]], (chisq.pval > 0.9))$term), collapse=",")
#head(orderBy(~-chisq.pval+freq+term, glb_post_stem_words_terms_df_lst[[txtFeat]]), 5)
#glbObsAll[glb_post_stem_words_terms_mtrx_lst[[txtFeat]][, 68] > 0, glbFeatsText]
#orderBy(~term, glb_post_stem_words_terms_df_lst[[txtFeat]][grep("^m", glb_post_stem_words_terms_df_lst[[txtFeat]]$term), ])

# To identify terms with high nzv.freqRatio
#summary(glb_post_stem_words_terms_df_lst[[txtFeat]]$nzv.freqRatio)
#paste0(sort(setdiff(subset(glb_post_stem_words_terms_df_lst[[txtFeat]], (nzv.freqRatio >= glbFeatsNzvFreqMax) & (freq < 10) & (chisq.pval >= 0.05))$term, c( "128gb","3g","4g","gold","ipad1","ipad3","ipad4","ipadair2","ipadmini2","manufactur","spacegray","sprint","tmobil","verizon","wifion"))), collapse=",")

# To identify obs with a txt term
#tail(orderBy(~-freq+term, glb_post_stop_words_terms_df_lst[[txtFeat]]), 20)
#mydspObs(list(descr.my.contains="non"), cols=c("color", "carrier", "cellular", "storage"))
#grep("ever", dimnames(terms_stop_mtrx)$Terms)
#which(terms_stop_mtrx[, grep("ipad", dimnames(terms_stop_mtrx)$Terms)] > 0)
#glbObsAll[which(terms_stop_mtrx[, grep("16", dimnames(terms_stop_mtrx)$Terms)[1]] > 0), c(glbFeatsCategory, "storage", txtFeat)]

# Text Processing Step: screen for names # Move to glbFeatsText specs section in order of text processing steps
# glbFeatsText[["<txtFeat>"]]$names <- myreplacePunctuation(str_to_lower(gsub(" ", "", c(NULL
#         # Person names for names screening
#         ,<comma-separated-list>
#         
#         # Company names
#         ,<comma-separated-list>
#                     
#         # Product names
#         ,<comma-separated-list>
#     ))))

# glbFeatsText[["<txtFeat>"]]$rareWords <- myreplacePunctuation(str_to_lower(gsub(" ", "", c(NULL
#         # Words not in SCOWL db
#         ,<comma-separated-list>
#     ))))

# To identify char vectors post glbFeatsTextMap
#grep("six(.*)hour", glb_txt_chr_lst[[txtFeat]], ignore.case = TRUE, value = TRUE)
#grep("[S|s]ix(.*)[H|h]our", glb_txt_chr_lst[[txtFeat]], value = TRUE)

# To identify whether terms shd be synonyms
#orderBy(~term, glb_post_stop_words_terms_df_lst[[txtFeat]][grep("^moder", glb_post_stop_words_terms_df_lst[[txtFeat]]$term), ])
# term_row_df <- glb_post_stop_words_terms_df_lst[[txtFeat]][grep("^came$", glb_post_stop_words_terms_df_lst[[txtFeat]]$term), ]
# 
# cor(glb_post_stop_words_terms_mtrx_lst[[txtFeat]][glbObsAll$.lcn == "Fit", term_row_df$pos], glbObsTrn[, glb_rsp_var], use="pairwise.complete.obs")

# To identify which stopped words are "close" to a txt term
#sort(glbFeatsCluster)

# Text Processing Step: stemDocument
# To identify stemmed txt terms
#glb_post_stop_words_terms_df_lst[[txtFeat]][grep("^la$", glb_post_stop_words_terms_df_lst[[txtFeat]]$term), ]
#orderBy(~term, glb_post_stem_words_terms_df_lst[[txtFeat]][grep("^con", glb_post_stem_words_terms_df_lst[[txtFeat]]$term), ])
#glbObsAll[which(terms_stem_mtrx[, grep("use", dimnames(terms_stem_mtrx)$Terms)[[1]]] > 0), c(glbFeatsId, "productline", txtFeat)]
#glbObsAll[which(TfIdf_stem_mtrx[, 191] > 0), c(glbFeatsId, glbFeatsCategory, txtFeat)]
#glbObsAll[which(glb_post_stop_words_terms_mtrx_lst[[txtFeat]][, 6165] > 0), c(glbFeatsId, glbFeatsCategory, txtFeat)]
#which(glbObsAll$UniqueID %in% c(11915, 11926, 12198))

# Text Processing Step: mycombineSynonyms
#   To identify which terms are associated with not -> combine "could not" & "couldn't"
#findAssocs(glb_full_DTM_lst[[txtFeat]], "not", 0.05)
#   To identify which synonyms should be combined
#orderBy(~term, glb_post_stem_words_terms_df_lst[[txtFeat]][grep("^c", glb_post_stem_words_terms_df_lst[[txtFeat]]$term), ])
chk_comb_cor <- function(syn_lst) {
#     cor(terms_stem_mtrx[glbObsAll$.src == "Train", grep("^(damag|dent|ding)$", dimnames(terms_stem_mtrx)[[2]])], glbObsTrn[, glb_rsp_var], use="pairwise.complete.obs")
    print(subset(glb_post_stem_words_terms_df_lst[[txtFeat]], term %in% syn_lst$syns))
    print(subset(get_corpus_terms(tm_map(glbFeatsTextCorpus[[txtFeat]], mycombineSynonyms, list(syn_lst), lazy=FALSE)), term == syn_lst$word))
#     cor(terms_stop_mtrx[glbObsAll$.src == "Train", grep("^(damage|dent|ding)$", dimnames(terms_stop_mtrx)[[2]])], glbObsTrn[, glb_rsp_var], use="pairwise.complete.obs")
#     cor(rowSums(terms_stop_mtrx[glbObsAll$.src == "Train", grep("^(damage|dent|ding)$", dimnames(terms_stop_mtrx)[[2]])]), glbObsTrn[, glb_rsp_var], use="pairwise.complete.obs")
}
#chk_comb_cor(syn_lst=list(word="cabl",  syns=c("cabl", "cord")))
#chk_comb_cor(syn_lst=list(word="damag",  syns=c("damag", "dent", "ding")))
#chk_comb_cor(syn_lst=list(word="dent",  syns=c("dent", "ding")))
#chk_comb_cor(syn_lst=list(word="use",  syns=c("use", "usag")))

glbFeatsTextSynonyms <- list()
# list parsed to collect glbFeatsText[[<txtFeat>]]$vldTerms
# glbFeatsTextSynonyms[["Hdln.my"]] <- list(NULL
#     # people in places
#     , list(word = "australia", syns = c("australia", "australian"))
#     , list(word = "italy", syns = c("italy", "Italian"))
#     , list(word = "newyork", syns = c("newyork", "newyorker"))    
#     , list(word = "Pakistan", syns = c("Pakistan", "Pakistani"))    
#     , list(word = "peru", syns = c("peru", "peruvian"))
#     , list(word = "qatar", syns = c("qatar", "qatari"))
#     , list(word = "scotland", syns = c("scotland", "scotish"))
#     , list(word = "Shanghai", syns = c("Shanghai", "Shanzhai"))    
#     , list(word = "venezuela", syns = c("venezuela", "venezuelan"))    
# 
#     # companies - needs to be data dependent 
#     #   - e.g. ensure BNP in this experiment/feat always refers to BNPParibas
#         
#     # general synonyms
#     , list(word = "Create", syns = c("Create","Creator")) 
#     , list(word = "cute", syns = c("cute","cutest"))     
#     , list(word = "Disappear", syns = c("Disappear","Fadeout"))     
#     , list(word = "teach", syns = c("teach", "taught"))     
#     , list(word = "theater",  syns = c("theater", "theatre", "theatres")) 
#     , list(word = "understand",  syns = c("understand", "understood"))    
#     , list(word = "weak",  syns = c("weak", "weaken", "weaker", "weakest"))
#     , list(word = "wealth",  syns = c("wealth", "wealthi"))    
#     
#     # custom synonyms (phrases)
#     
#     # custom synonyms (names)
#                                       )
#glbFeatsTextSynonyms[["<txtFeat>"]] <- list(NULL
#     , list(word="<stem1>",  syns=c("<stem1>", "<stem1_2>"))
#                                       )

for (txtFeat in names(glbFeatsTextSynonyms))
    for (entryIx in 1:length(glbFeatsTextSynonyms[[txtFeat]])) {
        glbFeatsTextSynonyms[[txtFeat]][[entryIx]]$word <-
            str_to_lower(glbFeatsTextSynonyms[[txtFeat]][[entryIx]]$word)
        glbFeatsTextSynonyms[[txtFeat]][[entryIx]]$syns <-
            str_to_lower(glbFeatsTextSynonyms[[txtFeat]][[entryIx]]$syns)        
    }        

glbFeatsTextSeed <- 181
# tm options include: check tm::weightSMART 
glb_txt_terms_control <- list( # Gather model performance & run-time stats
                    # weighting = function(x) weightSMART(x, spec = "nnn")
                    # weighting = function(x) weightSMART(x, spec = "lnn")
                    # weighting = function(x) weightSMART(x, spec = "ann")
                    # weighting = function(x) weightSMART(x, spec = "bnn")
                    # weighting = function(x) weightSMART(x, spec = "Lnn")
                    # 
                    weighting = function(x) weightSMART(x, spec = "ltn") # default
                    # weighting = function(x) weightSMART(x, spec = "lpn")                    
                    # 
                    # weighting = function(x) weightSMART(x, spec = "ltc")                    
                    # 
                    # weighting = weightBin 
                    # weighting = weightTf 
                    # weighting = weightTfIdf # : default
                # termFreq selection criteria across obs: tm default: list(global=c(1, Inf))
                    , bounds = list(global = c(1, Inf)) 
                # wordLengths selection criteria: tm default: c(3, Inf)
                    , wordLengths = c(1, Inf) 
                              ) 

glb_txt_cor_var <- glb_rsp_var # : default # or c(<feat>)

# select one from c("union.top.val.cor", "top.cor", "top.val", default: "top.chisq", "sparse")
glbFeatsTextFilter <- "top.chisq" 
glbFeatsTextTermsMax <- rep(10, length(glbFeatsText)) # :default
names(glbFeatsTextTermsMax) <- names(glbFeatsText)

# Text Processing Step: extractAssoc
glbFeatsTextAssocCor <- rep(1, length(glbFeatsText)) # :default 
names(glbFeatsTextAssocCor) <- names(glbFeatsText)

# Remember to use stemmed terms
glb_important_terms <- list()

# Text Processing Step: extractPatterns (ngrams)
glbFeatsTextPatterns <- list()
#glbFeatsTextPatterns[[<txtFeat>>]] <- list()
#glbFeatsTextPatterns[[<txtFeat>>]] <- c(metropolitan.diary.colon = "Metropolitan Diary:")

# Have to set it even if it is not used
# Properties:
#   numrows(glb_feats_df) << numrows(glbObsFit
#   Select terms that appear in at least 0.2 * O(FP/FN(glbObsOOB)) ???
#       numrows(glbObsOOB) = 1.1 * numrows(glbObsNew) ???
glb_sprs_thresholds <- NULL # or c(<txtFeat1> = 0.988, <txtFeat2> = 0.970, <txtFeat3> = 0.970)

glbFctrMaxUniqVals <- 20 # default: 20
glb_impute_na_data <- FALSE # or TRUE
glb_mice_complete.seed <- 144 # or any integer

glbFeatsCluster <- c("YOB.Age.fctr", "Gender.fctr", "Income.fctr", 
                     # "Hhold.fctr",
                     "Edn.fctr",
                     paste(grep("^Q.", glbFeatsExclude, value = TRUE), "fctr", sep = ".")) # NULL : default or c("<feat1>", "<feat2>")
# glbFeatsCluster <- grep(paste0("[", 
#                         toupper(paste0(substr(glbFeatsText, 1, 1), collapse = "")),
#                                       "]\\.[PT]\\."), 
#                                names(glbObsAll), value = TRUE)

glb_cluster.seed <- 189 # or any integer
glbClusterEntropyVar <- NULL # c(glb_rsp_var, as.factor(cut(glb_rsp_var, 3)), default: NULL)
glbFeatsClusterVarsExclude <- FALSE # default FALSE

glb_interaction_only_feats <- NULL # : default or c(<parent_feat> = "<child_feat>")

glbFeatsNzvFreqMax <- 19 # 19 : caret default
glbFeatsNzvUniqMin <- 10 # 10 : caret default

glbRFESizes <- list()
#glbRFESizes[["mdlFamily"]] <- c(4, 8, 16, 32, 64, 67, 68, 69) # Accuracy@69/70 = 0.8258

glbObsFitOutliers <- list()
# If outliers.n >= 10; consider concatenation of interaction vars
# glbObsFitOutliers[["<mdlFamily>"]] <- c(NULL
#     is.na(.rstudent)
#     max(.rstudent)
#     is.na(.dffits)
#     .hatvalues >= 0.99        
#     -38,167,642 < minmax(.rstudent) < 49,649,823    
#     , <comma-separated-<glbFeatsId>>
#                                     )
glbObsTrnOutliers <- list()
glbObsTrnOutliers[["Final"]] <- union(glbObsFitOutliers[["All.X"]],
                                c(NULL
                                ))

# influence.measures: car::outlier; rstudent; dffits; hatvalues; dfbeta; dfbetas
#mdlId <- "All.X##rcv#glm"; obs_df <- fitobs_df
#mdlId <- "RFE.X.glm"; obs_df <- fitobs_df
#mdlId <- "Final.glm"; obs_df <- trnobs_df
#mdlId <- "CSM2.X.glm"; obs_df <- fitobs_df
#print(outliers <- car::outlierTest(glb_models_lst[[mdlId]]$finalModel))
#mdlIdFamily <- paste0(head(unlist(str_split(mdlId, "\\.")), -1), collapse="."); obs_df <- dplyr::filter_(obs_df, interp(~(!(var %in% glbObsFitOutliers[[mdlIdFamily]])), var = as.name(glbFeatsId))); model_diags_df <- cbind(obs_df, data.frame(.rstudent=stats::rstudent(glb_models_lst[[mdlId]]$finalModel)), data.frame(.dffits=stats::dffits(glb_models_lst[[mdlId]]$finalModel)), data.frame(.hatvalues=stats::hatvalues(glb_models_lst[[mdlId]]$finalModel)));print(summary(model_diags_df[, c(".rstudent",".dffits",".hatvalues")])); table(cut(model_diags_df$.hatvalues, breaks=c(0.00, 0.98, 0.99, 1.00)))

#print(subset(model_diags_df, is.na(.rstudent))[, glbFeatsId])
#print(model_diags_df[which.max(model_diags_df$.rstudent), ])
#print(subset(model_diags_df, is.na(.dffits))[, glbFeatsId])
#print(model_diags_df[which.min(model_diags_df$.dffits), ])
#print(subset(model_diags_df, .hatvalues > 0.99)[, glbFeatsId])
#dffits_df <- merge(dffits_df, outliers_df, by="row.names", all.x=TRUE); row.names(dffits_df) <- dffits_df$Row.names; dffits_df <- subset(dffits_df, select=-Row.names)
#dffits_df <- merge(dffits_df, glbObsFit, by="row.names", all.x=TRUE); row.names(dffits_df) <- dffits_df$Row.names; dffits_df <- subset(dffits_df, select=-Row.names)
#subset(dffits_df, !is.na(.Bonf.p))

#mdlId <- "CSM.X.glm"; vars <- myextract_actual_feats(row.names(orderBy(reformulate(c("-", paste0(mdlId, ".imp"))), myget_feats_imp(glb_models_lst[[mdlId]])))); 
#model_diags_df <- glb_get_predictions(model_diags_df, mdlId, glb_rsp_var)
#obs_ix <- row.names(model_diags_df) %in% names(outliers$rstudent)[1]
#obs_ix <- which(is.na(model_diags_df$.rstudent))
#obs_ix <- which(is.na(model_diags_df$.dffits))
#myplot_parcoord(obs_df=model_diags_df[, c(glbFeatsId, glbFeatsCategory, ".rstudent", ".dffits", ".hatvalues", glb_rsp_var, paste0(glb_rsp_var, mdlId), vars[1:min(20, length(vars))])], obs_ix=obs_ix, id_var=glbFeatsId, category_var=glbFeatsCategory)

#model_diags_df[row.names(model_diags_df) %in% names(outliers$rstudent)[c(1:2)], ]
#ctgry_diags_df <- model_diags_df[model_diags_df[, glbFeatsCategory] %in% c("Unknown#0"), ]
#myplot_parcoord(obs_df=ctgry_diags_df[, c(glbFeatsId, glbFeatsCategory, ".rstudent", ".dffits", ".hatvalues", glb_rsp_var, "startprice.log10.predict.RFE.X.glmnet", indepVar[1:20])], obs_ix=row.names(ctgry_diags_df) %in% names(outliers$rstudent)[1], id_var=glbFeatsId, category_var=glbFeatsCategory)
#table(glbObsFit[model_diags_df[, glbFeatsCategory] %in% c("iPad1#1"), "startprice.log10.cut.fctr"])
#glbObsFit[model_diags_df[, glbFeatsCategory] %in% c("iPad1#1"), c(glbFeatsId, "startprice")]

# No outliers & .dffits == NaN
#myplot_parcoord(obs_df=model_diags_df[, c(glbFeatsId, glbFeatsCategory, glb_rsp_var, "startprice.log10.predict.RFE.X.glmnet", indepVar[1:10])], obs_ix=seq(1:nrow(model_diags_df))[is.na(model_diags_df$.dffits)], id_var=glbFeatsId, category_var=glbFeatsCategory)

# Modify mdlId to (build & extract) "<FamilyId>#<Fit|Trn>#<caretMethod>#<preProc1.preProc2>#<samplingMethod>"
glb_models_lst <- list(); glb_models_df <- data.frame()

# Add xgboost algorithm

# Regression
if (glb_is_regression) {
    glbMdlMethods <- c(NULL
        # deterministic
            #, "lm", # same as glm
            , "glm", "bayesglm", "glmnet"
            , "rpart"
        # non-deterministic
            , "gbm", "rf" 
        # Unknown
            , "nnet" , "avNNet" # runs 25 models per cv sample for tunelength=5
            , "svmLinear", "svmLinear2"
            , "svmPoly" # runs 75 models per cv sample for tunelength=5
            , "svmRadial" 
            , "earth"
            , "bagEarth" # Takes a long time
            ,"xgbLinear","xgbTree"
        )
} else
# Classification - Add ada (auto feature selection)
    if (glb_is_binomial)
        glbMdlMethods <- c(NULL
        # deterministic                     
            , "bagEarth" # Takes a long time        
            , "glm", "bayesglm", "glmnet"
            , "nnet"
            , "rpart"
        # non-deterministic        
            , "gbm"
            , "avNNet" # runs 25 models per cv sample for tunelength=5      
            , "rf"
        # Unknown
            , "lda", "lda2"
                # svm models crash when predict is called -> internal to kernlab it should call predict without .outcome
            , "svmLinear", "svmLinear2"
            , "svmPoly" # runs 75 models per cv sample for tunelength=5
            , "svmRadial" 
            , "earth"
            ,"xgbLinear","xgbTree"
        ) else
        glbMdlMethods <- c(NULL
        # deterministic
            ,"glmnet"
        # non-deterministic 
            ,"rf"       
        # Unknown
            ,"gbm","rpart","xgbLinear","xgbTree"
        )

glbMdlFamilies <- list(); glb_mdl_feats_lst <- list()
# family: Choose from c("RFE.X", "CSM.X", "All.X", "Best.Interact")
#   methods: Choose from c(NULL, <method>, glbMdlMethods) 
#glbMdlFamilies[["RFE.X"]] <- c("glmnet", "glm") # non-NULL vector is mandatory
if (glb_is_classification && !glb_is_binomial)
    # glm does not work for multinomial
    glbMdlFamilies[["All.X"]] <- c("glmnet") else    
    glbMdlFamilies[["All.X"]] <- c("glmnet", "glm")

#glbMdlFamilies[["Best.Interact"]] <- "glmnet" # non-NULL vector is mandatory

# Check if interaction features make RFE better
# glbMdlFamilies[["CSM.X"]] <- setdiff(glbMdlMethods, c("lda", "lda2")) # crashing due to category:.clusterid ??? #c("glmnet", "glm") # non-NULL list is mandatory
# glb_mdl_feats_lst[["CSM.X"]] <- c(NULL
#     , <comma-separated-features-vector>
#                                   )
# dAFeats.CSM.X %<d-% c(NULL
#     # Interaction feats up to varImp(RFE.X.glmnet) >= 50
#     , <comma-separated-features-vector>
#     , setdiff(myextract_actual_feats(predictors(rfe_fit_results)), c(NULL
#                , <comma-separated-features-vector>
#                                                                       ))    
#                                   )
# glb_mdl_feats_lst[["CSM.X"]] <- "%<d-% dAFeats.CSM.X"

glbMdlFamilies[["Final"]] <- c(NULL) # NULL vector acceptable # c("glmnet", "glm")

glbMdlAllowParallel <- list()
#glbMdlAllowParallel[["Final##rcv#glmnet"]] <- FALSE
glbMdlAllowParallel[["All.X##rcv#glm"]] <- FALSE

# Check if tuning parameters make fit better; make it mdlFamily customizable ?
glbMdlTuneParams <- data.frame()
# When glmnet crashes at model$grid with error: ???
glmnetTuneParams <- rbind(data.frame()
                        ,data.frame(parameter = "alpha",  vals = "0.100 0.325 0.550 0.775 1.000")
                        ,data.frame(parameter = "lambda", vals = "9.342e-02")    
                        )
# glbMdlTuneParams <- myrbind_df(glbMdlTuneParams,
#                                cbind(data.frame(mdlId = "<mdlId>"),
#                                      glmnetTuneParams))

    #avNNet    
    #   size=[1] 3 5 7 9; decay=[0] 1e-04 0.001  0.01   0.1; bag=[FALSE]; RMSE=1.3300906 

    #bagEarth
    #   degree=1 [2] 3; nprune=64 128 256 512 [1024]; RMSE=0.6486663 (up)
# glbMdlTuneParams <- myrbind_df(glbMdlTuneParams, rbind(data.frame()
#     ,data.frame(method = "bagEarth", parameter = "nprune", vals = "256")
#     ,data.frame(method = "bagEarth", parameter = "degree", vals = "2")    
# ))

    #earth 
    #   degree=[1]; nprune=2  [9] 17 25 33; RMSE=0.1334478
    
    #gbm 
    #   shrinkage=0.05 [0.10] 0.15 0.20 0.25; n.trees=100 150 200 [250] 300; interaction.depth=[1] 2 3 4 5; n.minobsinnode=[10]; RMSE=0.2008313     
# glbMdlTuneParams <- myrbind_df(glbMdlTuneParams, rbind(data.frame()
#     ,data.frame(method = "gbm", parameter = "shrinkage", min = 0.05, max = 0.25, by = 0.05)
#     ,data.frame(method = "gbm", parameter = "n.trees", min = 100, max = 300, by = 50)
#     ,data.frame(method = "gbm", parameter = "interaction.depth", min = 1, max = 5, by = 1)
#     ,data.frame(method = "gbm", parameter = "n.minobsinnode", min = 10, max = 10, by = 10)
#     #seq(from=0.05,  to=0.25, by=0.05)
# ))

    #glmnet
    #   alpha=0.100 [0.325] 0.550 0.775 1.000; lambda=0.0005232693 0.0024288010 0.0112734954 [0.0523269304] 0.2428800957; RMSE=0.6164891
# glbMdlTuneParams <- myrbind_df(glbMdlTuneParams, rbind(data.frame()
#     ,data.frame(method = "glmnet", parameter = "alpha", vals = "0.550 0.775 0.8875 0.94375 1.000")
#     ,data.frame(method = "glmnet", parameter = "lambda", vals = "9.858855e-05 0.0001971771 0.0009152152 0.0042480525 0.0197177130")    
# ))

    #nnet    
    #   size=3 5 [7] 9 11; decay=0.0001 0.001 0.01 [0.1] 0.2; RMSE=0.9287422
# glbMdlTuneParams <- myrbind_df(glbMdlTuneParams, rbind(data.frame()
#     ,data.frame(method = "nnet", parameter = "size", vals = "3 5 7 9 11")
#     ,data.frame(method = "nnet", parameter = "decay", vals = "0.0001 0.0010 0.0100 0.1000 0.2000")    
# ))

    #rf # Don't bother; results are not deterministic
    #       mtry=2  35  68 [101] 134; RMSE=0.1339974
# glbMdlTuneParams <- myrbind_df(glbMdlTuneParams, rbind(data.frame()
#     ,data.frame(method = "rf", parameter = "mtry", vals = "2 5 9 13 17")
# ))

    #rpart 
    #   cp=0.020 [0.025] 0.030 0.035 0.040; RMSE=0.1770237
# glbMdlTuneParams <- myrbind_df(glbMdlTuneParams, rbind(data.frame()    
#     ,data.frame(method = "rpart", parameter = "cp", vals = "0.004347826 0.008695652 0.017391304 0.021739130 0.034782609")
# ))
    
    #svmLinear
    #   C=0.01 0.05 [0.10] 0.50 1.00 2.00 3.00 4.00; RMSE=0.1271318; 0.1296718
# glbMdlTuneParams <- myrbind_df(glbMdlTuneParams, rbind(data.frame()
#     ,data.frame(method = "svmLinear", parameter = "C", vals = "0.01 0.05 0.1 0.5 1")
# ))

    #svmLinear2    
    #   cost=0.0625 0.1250 [0.25] 0.50 1.00; RMSE=0.1276354 
# glbMdlTuneParams <- myrbind_df(glbMdlTuneParams, rbind(data.frame()
#     ,data.frame(method = "svmLinear2", parameter = "cost", vals = "0.0625 0.125 0.25 0.5 1")
# ))

    #svmPoly    
    #   degree=[1] 2 3 4 5; scale=0.01 0.05 [0.1] 0.5 1; C=0.50 1.00 [2.00] 3.00 4.00; RMSE=0.1276130
# glbMdlTuneParams <- myrbind_df(glbMdlTuneParams, rbind(data.frame()
#     ,data.frame(method="svmPoly", parameter="degree", min=1, max=5, by=1) #seq(1, 5, 1)
#     ,data.frame(method="svmPoly", parameter="scale", vals="0.01, 0.05, 0.1, 0.5, 1")
#     ,data.frame(method="svmPoly", parameter="C", vals="0.50, 1.00, 2.00, 3.00, 4.00")    
# ))

    #svmRadial
    #   sigma=[0.08674323]; C=0.25 0.50 1.00 [2.00] 4.00; RMSE=0.1614957
    
#glb2Sav(); all.equal(sav_models_df, glb_models_df)
    
glb_preproc_methods <- NULL
#     c("YeoJohnson", "center.scale", "range", "pca", "ica", "spatialSign")

# Baseline prediction model feature(s)
glb_Baseline_mdl_var <- NULL # or c("<feat>")

glbMdlMetric_terms <- NULL # or matrix(c(
#                               0,1,2,3,4,
#                               2,0,1,2,3,
#                               4,2,0,1,2,
#                               6,4,2,0,1,
#                               8,6,4,2,0
#                           ), byrow=TRUE, nrow=5)
glbMdlMetricSummary <- NULL # or "<metric_name>"
glbMdlMetricMaximize <- NULL # or FALSE (TRUE is not the default for both classification & regression) 
glbMdlMetricSummaryFn <- NULL # or function(data, lev=NULL, model=NULL) {
#     confusion_mtrx <- t(as.matrix(confusionMatrix(data$pred, data$obs)))
#     #print(confusion_mtrx)
#     #print(confusion_mtrx * glbMdlMetric_terms)
#     metric <- sum(confusion_mtrx * glbMdlMetric_terms) / nrow(data)
#     names(metric) <- glbMdlMetricSummary
#     return(metric)
# }

glbMdlCheckRcv <- FALSE # Turn it on when needed; otherwise takes long time
glb_rcv_n_folds <- 3 # or NULL
glb_rcv_n_repeats <- 3 # or NULL

glb_clf_proba_threshold <- NULL # 0.5

# Model selection criteria
if (glb_is_regression)
    glbMdlMetricsEval <- c("min.RMSE.OOB", "max.R.sq.OOB", "max.Adj.R.sq.fit", "min.RMSE.fit")
    #glbMdlMetricsEval <- c("min.RMSE.fit", "max.R.sq.fit", "max.Adj.R.sq.fit")    
if (glb_is_classification) {
    if (glb_is_binomial)
        glbMdlMetricsEval <- 
            c("max.Accuracy.OOB", "max.AUCROCR.OOB", "max.AUCpROC.OOB", "min.aic.fit", "max.Accuracy.fit") else        
        glbMdlMetricsEval <- c("max.Accuracy.OOB", "max.Kappa.OOB")
}

# select from NULL [no ensemble models], "auto" [all models better than MFO or Baseline], c(mdl_ids in glb_models_lst) [Typically top-rated models in auto]
glb_mdl_ensemble <- NULL
#     "%<d-% setdiff(mygetEnsembleAutoMdlIds(), 'CSM.X.rf')" 
#     c(<comma-separated-mdlIds>
#      )

# Only for classifications; for regressions remove "(.*)\\.prob" form the regex
# tmp_fitobs_df <- glbObsFit[, grep(paste0("^", gsub(".", "\\.", mygetPredictIds$value, fixed = TRUE), "CSM\\.X\\.(.*)\\.prob"), names(glbObsFit), value = TRUE)]; cor_mtrx <- cor(tmp_fitobs_df); cor_vctr <- sort(cor_mtrx[row.names(orderBy(~-Overall, varImp(glb_models_lst[["Ensemble.repeatedcv.glmnet"]])$imp))[1], ]); summary(cor_vctr); cor_vctr
#ntv.glm <- glm(reformulate(indepVar, glb_rsp_var), family = "binomial", data = glbObsFit)
#step.glm <- step(ntv.glm)

glbMdlSelId <- "All.X##rcv#glmnet" #select from c(NULL, "All.X##rcv#glmnet", "RFE.X##rcv#glmnet", <mdlId>)
glbMdlFinId <- NULL #select from c(NULL, glbMdlSelId)

glb_dsp_cols <- c(".pos", glbFeatsId, glbFeatsCategory, glb_rsp_var
#               List critical cols excl. above
                  )

# Output specs
# lclgetfltout_df <- function(obsOutFinDf) {
#     require(tidyr)
#     obsOutFinDf <- obsOutFinDf %>%
#         tidyr::separate("ImageId.x.y", c(".src", ".pos", "x", "y"), 
#                         sep = "#", remove = TRUE, extra = "merge")
#     # mnm prefix stands for max_n_mean
#     mnmout_df <- obsOutFinDf %>%
#         dplyr::group_by(.pos) %>%
#         #dplyr::top_n(1, Probability1) %>% # Score = 3.9426         
#         #dplyr::top_n(2, Probability1) %>% # Score = ???; weighted = 3.94254;         
#         #dplyr::top_n(3, Probability1) %>% # Score = 3.9418; weighted = 3.94169; 
#         dplyr::top_n(4, Probability1) %>% # Score = ???; weighted = 3.94149;        
#         #dplyr::top_n(5, Probability1) %>% # Score = 3.9421; weighted = 3.94178
#     
#         # dplyr::summarize(xMeanN = mean(as.numeric(x)), yMeanN = mean(as.numeric(y)))
#         # dplyr::summarize(xMeanN = weighted.mean(as.numeric(x), Probability1), yMeanN = mean(as.numeric(y)))
#         # dplyr::summarize(xMeanN = weighted.mean(as.numeric(x), c(Probability1, 0.2357323, 0.2336925)), yMeanN = mean(as.numeric(y)))    
#         # dplyr::summarize(xMeanN = weighted.mean(as.numeric(x), c(Probability1)), yMeanN = mean(as.numeric(y)))
#         dplyr::summarize(xMeanN = weighted.mean(as.numeric(x), c(Probability1)), 
#                          yMeanN = weighted.mean(as.numeric(y), c(Probability1)))  
#     
#     maxout_df <- obsOutFinDf %>%
#         dplyr::group_by(.pos) %>%
#         dplyr::summarize(maxProb1 = max(Probability1))
#     fltout_df <- merge(maxout_df, obsOutFinDf, 
#                        by.x = c(".pos", "maxProb1"), by.y = c(".pos", "Probability1"),
#                        all.x = TRUE)
#     fmnout_df <- merge(fltout_df, mnmout_df, 
#                        by.x = c(".pos"), by.y = c(".pos"),
#                        all.x = TRUE)
#     return(fmnout_df)
# }
glbObsOut <- list(NULL
        # glbFeatsId will be the first output column, by default
        ,vars = list()
#         ,mapFn = function(obsOutFinDf) {
#                   }
                  )
#obsOutFinDf <- savobsOutFinDf
# glbObsOut$mapFn <- function(obsOutFinDf) {
#     txfout_df <- dplyr::select(obsOutFinDf, -.pos.y) %>%
#         dplyr::mutate(
#             lunch     = levels(glbObsTrn[, "lunch"    ])[
#                        round(mean(as.numeric(glbObsTrn[, "lunch"    ])), 0)],
#             dinner    = levels(glbObsTrn[, "dinner"   ])[
#                        round(mean(as.numeric(glbObsTrn[, "dinner"   ])), 0)],
#             reserve   = levels(glbObsTrn[, "reserve"  ])[
#                        round(mean(as.numeric(glbObsTrn[, "reserve"  ])), 0)],
#             outdoor   = levels(glbObsTrn[, "outdoor"  ])[
#                        round(mean(as.numeric(glbObsTrn[, "outdoor"  ])), 0)],
#             expensive = levels(glbObsTrn[, "expensive"])[
#                        round(mean(as.numeric(glbObsTrn[, "expensive"])), 0)],
#             liquor    = levels(glbObsTrn[, "liquor"   ])[
#                        round(mean(as.numeric(glbObsTrn[, "liquor"   ])), 0)],
#             table     = levels(glbObsTrn[, "table"    ])[
#                        round(mean(as.numeric(glbObsTrn[, "table"    ])), 0)],
#             classy    = levels(glbObsTrn[, "classy"   ])[
#                        round(mean(as.numeric(glbObsTrn[, "classy"   ])), 0)],
#             kids      = levels(glbObsTrn[, "kids"     ])[
#                        round(mean(as.numeric(glbObsTrn[, "kids"     ])), 0)]
#                       )
#     
#     print("ObsNew output class tables:")
#     print(sapply(c("lunch","dinner","reserve","outdoor",
#                    "expensive","liquor","table",
#                    "classy","kids"), 
#                  function(feat) table(txfout_df[, feat], useNA = "ifany")))
#     
#     txfout_df <- txfout_df %>%
#         dplyr::mutate(labels = "") %>%
#         dplyr::mutate(labels = 
#     ifelse(lunch     != "-1", paste(labels, lunch    ), labels)) %>%
#         dplyr::mutate(labels = 
#     ifelse(dinner    != "-1", paste(labels, dinner   ), labels)) %>%
#         dplyr::mutate(labels = 
#     ifelse(reserve   != "-1", paste(labels, reserve  ), labels)) %>%
#         dplyr::mutate(labels = 
#     ifelse(outdoor   != "-1", paste(labels, outdoor  ), labels)) %>%
#         dplyr::mutate(labels =         
#     ifelse(expensive != "-1", paste(labels, expensive), labels)) %>%
#         dplyr::mutate(labels =         
#     ifelse(liquor    != "-1", paste(labels, liquor   ), labels)) %>%
#         dplyr::mutate(labels =         
#     ifelse(table     != "-1", paste(labels, table    ), labels)) %>%
#         dplyr::mutate(labels =         
#     ifelse(classy    != "-1", paste(labels, classy   ), labels)) %>%
#         dplyr::mutate(labels =         
#     ifelse(kids      != "-1", paste(labels, kids     ), labels)) %>%
#         dplyr::select(business_id, labels)
#     return(txfout_df)
# }
#if (!is.null(glbObsOut$mapFn)) obsOutFinDf <- glbObsOut$mapFn(obsOutFinDf); print(head(obsOutFinDf))

glb_out_obs <- NULL # select from c(NULL : default to "new", "all", "new", "trn")

if (glb_is_classification && glb_is_binomial) {
    # glbObsOut$vars[["Probability1"]] <- 
    #     "%<d-% glbObsNew[, mygetPredictIds(glb_rsp_var, glbMdlFinId)$prob]" 
    # glbObsOut$vars[[glb_rsp_var_raw]] <-
    #     "%<d-% glb_map_rsp_var_to_raw(glbObsNew[,
    #                                         mygetPredictIds(glb_rsp_var, glbMdlFinId)$value])"
    glbObsOut$vars[["Predictions"]] <-
        "%<d-% glb_map_rsp_var_to_raw(glbObsNew[,
                                            mygetPredictIds(glb_rsp_var, glbMdlFinId)$value])"
} else {
#     glbObsOut$vars[[glbFeatsId]] <- 
#         "%<d-% as.integer(gsub('Test#', '', glbObsNew[, glbFeatsId]))"
    glbObsOut$vars[[glb_rsp_var]] <- 
        "%<d-% glbObsNew[, mygetPredictIds(glb_rsp_var, glbMdlFinId)$value]"
#     for (outVar in setdiff(glbFeatsExcludeLcl, glb_rsp_var_raw))
#         glbObsOut$vars[[outVar]] <- 
#             paste0("%<d-% mean(glbObsAll[, \"", outVar, "\"], na.rm = TRUE)")
}    
# glbObsOut$vars[[glb_rsp_var_raw]] <- glb_rsp_var_raw
# glbObsOut$vars[[paste0(head(unlist(strsplit(mygetPredictIds$value, "")), -1), collapse = "")]] <-

glbOutStackFnames <- NULL #: default
    # c("ebayipads_txt_assoc1_out_bid1_stack.csv") # manual stack
    # c("ebayipads_finmdl_bid1_out_nnet_1.csv") # universal stack

glbOut <- list(pfx = "Votes_Q_02_cluster_")
# lclImageSampleSeed <- 129
glbOutDataVizFname <- NULL # choose from c(NULL, "<projectId>_obsall.csv")


glbChunks <- list(labels = c("set_global_options_wd","set_global_options"
    ,"import.data","inspect.data","scrub.data","transform.data"
    ,"extract.features"
        ,"extract.features.datetime","extract.features.image","extract.features.price"
        ,"extract.features.text","extract.features.string"  
        ,"extract.features.end"
    ,"manage.missing.data","cluster.data","partition.data.training","select.features"
    ,"fit.models_0","fit.models_1","fit.models_2","fit.models_3"
    ,"fit.data.training_0","fit.data.training_1"
    ,"predict.data.new"         
    ,"display.session.info"))
# To ensure that all chunks in this script are in glbChunks
if (!is.null(chkChunksLabels <- knitr::all_labels()) && # knitr::all_labels() doesn't work in console runs
    !identical(chkChunksLabels, glbChunks$labels)) {
    print(sprintf("setdiff(chkChunksLabels, glbChunks$labels): %s", 
                  setdiff(chkChunksLabels, glbChunks$labels)))    
    print(sprintf("setdiff(glbChunks$labels, chkChunksLabels): %s", 
                  setdiff(glbChunks$labels, chkChunksLabels)))    
}

glbChunks[["first"]] <- "cluster.data" #default: script will load envir from previous chunk
glbChunks[["last"]] <- NULL #NULL #default: script will save envir at end of this chunk 
glbChunks[["inpFilePathName"]] <- "data/Votes_Q_01_cnk_manage.missing.data.RData"
#mysavChunk(glbOut$pfx, glbChunks[["last"]]) # called from myevlChunk

# Inspect max OOB FP
#chkObsOOB <- subset(glbObsOOB, !label.fctr.All.X..rcv.glmnet.is.acc)
#chkObsOOBFP <- subset(chkObsOOB, label.fctr.All.X..rcv.glmnet == "left_eye_center") %>% dplyr::mutate(Probability1 = label.fctr.All.X..rcv.glmnet.prob) %>% select(-.src, -.pos, -x, -y) %>% lclgetfltout_df() %>% mutate(obj.distance = (((as.numeric(x) - left_eye_center_x.int) ^ 2) + ((as.numeric(y) - left_eye_center_y.int) ^ 2)) ^ 0.5) %>% dplyr::top_n(5, obj.distance) %>% dplyr::top_n(5, -patch.cor)
#
#newImgObs <- glbObsNew[(glbObsNew$ImageId == "Test#0001"), ]; print(newImgObs[which.max(newImgObs$label.fctr.Final..rcv.glmnet.prob), ])
#OOBImgObs <- glbObsOOB[(glbObsOOB$ImageId == "Train#0003"), ]; print(OOBImgObs[which.max(OOBImgObs$label.fctr.All.X..rcv.glmnet.prob), ])

#load("Votes_Q_02_cnk_extract.features.end.RData", verbose = TRUE)
#mygetImage(which(glbObsAll[, glbFeatsId] == "Train#0003"), names(glbFeatsImage)[1], plot = TRUE, featHighlight = c("left_eye_center_x", "left_eye_center_y"), ovrlHighlight = c(66, 35))

# Depict process
glb_analytics_pn <- petrinet(name = "glb_analytics_pn",
                        trans_df = data.frame(id = 1:6,
    name = c("data.training.all","data.new",
           "model.selected","model.final",
           "data.training.all.prediction","data.new.prediction"),
    x=c(   -5,-5,-15,-25,-25,-35),
    y=c(   -5, 5,  0,  0, -5,  5)
                        ),
                        places_df=data.frame(id=1:4,
    name=c("bgn","fit.data.training.all","predict.data.new","end"),
    x=c(   -0,   -20,                    -30,               -40),
    y=c(    0,     0,                      0,                 0),
    M0=c(   3,     0,                      0,                 0)
                        ),
                        arcs_df = data.frame(
    begin = c("bgn","bgn","bgn",        
            "data.training.all","model.selected","fit.data.training.all",
            "fit.data.training.all","model.final",    
            "data.new","predict.data.new",
            "data.training.all.prediction","data.new.prediction"),
    end   = c("data.training.all","data.new","model.selected",
            "fit.data.training.all","fit.data.training.all","model.final",
            "data.training.all.prediction","predict.data.new",
            "predict.data.new","data.new.prediction",
            "end","end")
                        ))
#print(ggplot.petrinet(glb_analytics_pn))
print(ggplot.petrinet(glb_analytics_pn) + coord_flip())

## Loading required package: grid

glb_analytics_avl_objs <- NULL

glb_chunks_df <- myadd_chunk(NULL, 
                             ifelse(is.null(glbChunks$first), "import.data", glbChunks$first))

##          label step_major step_minor label_minor   bgn end elapsed
## 1 cluster.data          1          0           0 9.146  NA      NA

Step `1.0: cluster data`

chunk option: eval=

Step `1.0: cluster data`

```{r extract.features.image, cache=FALSE, echo=FALSE, fig.height=5, fig.width=5, eval=myevlChunk(glbChunks, glbOut$pfx)}

Step `1.0: cluster data`

`{r cluster.data, cache=FALSE, echo=FALSE, eval=myevlChunk(glbChunks, glbOut$pfx)} #`{r cluster.data, cache=FALSE, echo=FALSE, eval=myevlChunk(glbChunks, glbOut$pfx, keep = c(“glbFeatsCategory”,“glb_dsp_cols”))}

## Loading required package: proxy

## 
## Attaching package: 'proxy'

## The following objects are masked from 'package:stats':
## 
##     as.dist, dist

## The following object is masked from 'package:base':
## 
##     as.matrix

## Loading required package: dynamicTreeCut

## Loading required package: entropy

## Loading required package: tidyr

## Loading required package: ggdendro

## [1] "Clustering features: "
##               abs.cor.y
## Q98197.fctr  0.05493425
## Q113181.fctr 0.08087531
## Q115611.fctr 0.09044682
## Gender.fctr  0.10274009
## Q109244.fctr 0.12038125
## [1] "    .rnorm cor: -0.0078"
## [1] "  Clustering entropy measure: Party.fctr"
## [1] "glbObsAll Entropy: 0.6913"

## Loading required package: lazyeval

##   Hhold.fctr .clusterid Hhold.fctr.clusterid    R    D  .entropy .knt
## 1          N          1                  N_1  220  230 0.6929002  450
## 2        MKn          1                MKn_1  308  344 0.6916221  652
## 3        MKy          1                MKy_1  842  752 0.6915524 1594
## 4        PKn          1                PKn_1   49  131 0.5854566  180
## 5        PKy          1                PKy_1   26   35 0.6822232   61
## 6        SKn          1                SKn_1 1091 1340 0.6878923 2431
## 7        SKy          1                SKy_1   81  119 0.6749870  200
## [1] "glbObsAll$Hhold.fctr Entropy: 0.6859 (99.2186 pct)"

## [1] "Category: N"
## [1] "max distance(0.9799) pair:"
##      USER_ID Party.fctr Hhold.fctr YOB.Age.fctr Gender.fctr Income.fctr
## 2175    2711          D          N      (50,65]           N       >150K
## 3742    4664          D          N           NA           N           N
##      Edn.fctr Q124742.fctr Q124122.fctr Q123621.fctr Q123464.fctr
## 2175      Bcr           NA           NA           NA           NA
## 3742        N           NA          Yes           NA           NA
##      Q122771.fctr Q122770.fctr Q122769.fctr Q122120.fctr Q121700.fctr
## 2175           NA           NA           NA           NA           NA
## 3742           Pc          Yes          Yes           NA           NA
##      Q121699.fctr Q121011.fctr Q120978.fctr Q120650.fctr Q120472.fctr
## 2175           NA           NA           NA           NA           NA
## 3742           NA           No          Yes          Yes      Science
##      Q120379.fctr Q120194.fctr Q120014.fctr Q120012.fctr Q119851.fctr
## 2175           NA           NA           NA           NA           NA
## 3742          Yes  Study first           No           NA           No
##      Q119650.fctr Q119334.fctr Q118892.fctr Q118237.fctr Q118233.fctr
## 2175           NA           NA          Yes           NA           NA
## 3742       Giving          Yes           NA          Yes           No
##      Q118232.fctr Q118117.fctr   Q117193.fctr Q117186.fctr Q116797.fctr
## 2175           NA           NA             NA           NA           NA
## 3742           Pr           No Standard hours           NA           NA
##      Q116881.fctr Q116953.fctr Q116601.fctr Q116441.fctr Q116448.fctr
## 2175           NA           NA           NA           NA           NA
## 3742           NA           NA          Yes           NA           NA
##      Q116197.fctr Q115602.fctr Q115777.fctr Q115610.fctr Q115611.fctr
## 2175           NA           NA           NA           NA           NA
## 3742           NA           NA          End           NA           NA
##      Q115899.fctr Q115390.fctr Q115195.fctr Q114961.fctr Q114748.fctr
## 2175           NA           NA           NA           NA           NA
## 3742           Me          Yes           No           NA           NA
##      Q114517.fctr Q114386.fctr Q114152.fctr Q113992.fctr Q113583.fctr
## 2175           NA           NA          Yes           NA           NA
## 3742           No           NA           NA           NA           NA
##      Q113584.fctr Q113181.fctr Q112478.fctr Q112512.fctr Q112270.fctr
## 2175   Technology           NA           NA           NA           NA
## 3742           NA           NA           No          Yes           No
##      Q111848.fctr Q111580.fctr Q111220.fctr Q110740.fctr Q109367.fctr
## 2175          Yes   Supportive           NA           NA           NA
## 3742          Yes    Demanding           No           NA          Yes
##      Q109244.fctr Q108950.fctr Q108855.fctr Q108617.fctr Q108856.fctr
## 2175           NA           NA           NA           NA           NA
## 3742           NA     Cautious           NA           No           NA
##      Q108754.fctr Q108342.fctr Q108343.fctr Q107869.fctr Q107491.fctr
## 2175           NA           NA           NA           NA           NA
## 3742           NA    In-person           No          Yes           NA
##      Q106993.fctr Q106997.fctr Q106272.fctr Q106388.fctr Q106389.fctr
## 2175           NA           NA           NA           NA           NA
## 3742          Yes           Yy          Yes          Yes           No
##      Q106042.fctr Q105840.fctr Q105655.fctr Q104996.fctr Q103293.fctr
## 2175           NA           NA           NA           NA           NA
## 3742           NA           No          Yes          Yes           No
##      Q102906.fctr Q102674.fctr Q102687.fctr Q102289.fctr Q102089.fctr
## 2175          Yes           No           NA           NA           NA
## 3742           No          Yes           No          Yes           NA
##      Q101162.fctr Q101163.fctr Q101596.fctr Q100689.fctr Q100680.fctr
## 2175           NA           NA           NA           NA           NA
## 3742           NA           NA          Yes          Yes           No
##      Q100562.fctr Q100010.fctr Q99982.fctr Q99716.fctr Q99581.fctr
## 2175          Yes           NA          NA          NA          NA
## 3742          Yes          Yes        Nope          NA          No
##      Q99480.fctr Q98869.fctr Q98578.fctr Q98197.fctr Q98059.fctr
## 2175          NA          NA          NA          NA          NA
## 3742         Yes          NA          NA          NA         Yes
##      Q98078.fctr Q96024.fctr
## 2175          NA          NA
## 3742          No          NA
## [1] "min distance(0.9403) pair:"
##      USER_ID Party.fctr Hhold.fctr YOB.Age.fctr Gender.fctr Income.fctr
## 1286    1591          R          N      (15,20]           M           N
## 2641    3283          R          N      (15,20]           M           N
##      Edn.fctr Q124742.fctr Q124122.fctr Q123621.fctr Q123464.fctr
## 1286      K12           NA           NA           NA           NA
## 2641        N           NA           NA           NA           NA
##      Q122771.fctr Q122770.fctr Q122769.fctr Q122120.fctr Q121700.fctr
## 1286           NA           NA           NA           NA           NA
## 2641           NA           NA           NA           NA           NA
##      Q121699.fctr Q121011.fctr Q120978.fctr Q120650.fctr Q120472.fctr
## 1286           NA           NA           NA           NA           NA
## 2641           NA           NA           NA           NA           NA
##      Q120379.fctr Q120194.fctr Q120014.fctr Q120012.fctr Q119851.fctr
## 1286           NA           NA           NA           NA           NA
## 2641           NA           NA           NA           NA           NA
##      Q119650.fctr Q119334.fctr Q118892.fctr Q118237.fctr Q118233.fctr
## 1286           NA           NA           NA           NA           NA
## 2641           NA           NA           NA           NA           NA
##      Q118232.fctr Q118117.fctr Q117193.fctr Q117186.fctr Q116797.fctr
## 1286           NA           NA           NA           NA           NA
## 2641           NA           NA           NA           NA           NA
##      Q116881.fctr Q116953.fctr Q116601.fctr Q116441.fctr Q116448.fctr
## 1286           NA           NA           NA           NA           NA
## 2641           NA           NA           NA           NA           NA
##      Q116197.fctr Q115602.fctr Q115777.fctr Q115610.fctr Q115611.fctr
## 1286           NA           NA           NA           NA           NA
## 2641           NA           NA           NA           NA           NA
##      Q115899.fctr Q115390.fctr Q115195.fctr Q114961.fctr Q114748.fctr
## 1286           NA           NA           NA           NA           No
## 2641           NA           NA           NA           NA           NA
##      Q114517.fctr Q114386.fctr Q114152.fctr Q113992.fctr Q113583.fctr
## 1286           No   Mysterious           NA           NA           NA
## 2641           NA           NA           NA           NA           NA
##      Q113584.fctr Q113181.fctr Q112478.fctr Q112512.fctr Q112270.fctr
## 1286           NA          Yes           NA           NA           NA
## 2641           NA          Yes          Yes          Yes          Yes
##      Q111848.fctr Q111580.fctr Q111220.fctr Q110740.fctr Q109367.fctr
## 1286           NA           NA           NA           NA           NA
## 2641           NA           NA           NA           NA           NA
##      Q109244.fctr Q108950.fctr Q108855.fctr Q108617.fctr Q108856.fctr
## 1286           NA           NA           NA           NA           NA
## 2641           NA           NA           NA           NA           NA
##      Q108754.fctr Q108342.fctr Q108343.fctr Q107869.fctr Q107491.fctr
## 1286           NA           NA           NA           NA           NA
## 2641           NA           NA           NA           NA           NA
##      Q106993.fctr Q106997.fctr Q106272.fctr Q106388.fctr Q106389.fctr
## 1286           NA           NA           NA           NA           NA
## 2641           NA           NA           NA           NA           NA
##      Q106042.fctr Q105840.fctr Q105655.fctr Q104996.fctr Q103293.fctr
## 1286           NA           NA           NA           NA           NA
## 2641           NA           NA           NA           NA           NA
##      Q102906.fctr Q102674.fctr Q102687.fctr Q102289.fctr Q102089.fctr
## 1286           NA           NA           NA           NA           NA
## 2641           NA           NA           NA           NA           NA
##      Q101162.fctr Q101163.fctr Q101596.fctr Q100689.fctr Q100680.fctr
## 1286           NA           NA           NA           NA           NA
## 2641           NA           NA           NA           NA           NA
##      Q100562.fctr Q100010.fctr Q99982.fctr Q99716.fctr Q99581.fctr
## 1286           NA           NA          NA          NA          NA
## 2641           NA           NA          NA          NA          NA
##      Q99480.fctr Q98869.fctr Q98578.fctr Q98197.fctr Q98059.fctr
## 1286          NA          NA          NA          NA          NA
## 2641          NA          NA          NA          NA          NA
##      Q98078.fctr Q96024.fctr
## 1286          NA          NA
## 2641          NA          NA
## [1] "Category: MKn"
## [1] "max distance(0.9773) pair:"
##      USER_ID Party.fctr Hhold.fctr YOB.Age.fctr Gender.fctr Income.fctr
## 2328    2894          D        MKn      (65,90]           M     75-100K
## 4767    5946          R        MKn      (65,90]           N           N
##      Edn.fctr Q124742.fctr Q124122.fctr Q123621.fctr Q123464.fctr
## 2328        N           NA           No           No           No
## 4767      Msr           No           NA           NA           NA
##      Q122771.fctr Q122770.fctr Q122769.fctr Q122120.fctr Q121700.fctr
## 2328           Pc          Yes          Yes           No           No
## 4767           NA           NA           NA           NA           NA
##      Q121699.fctr Q121011.fctr Q120978.fctr Q120650.fctr Q120472.fctr
## 2328          Yes          Yes           No          Yes      Science
## 4767           NA           NA           NA           NA           NA
##      Q120379.fctr Q120194.fctr Q120014.fctr Q120012.fctr Q119851.fctr
## 2328           No  Study first          Yes          Yes          Yes
## 4767           NA           NA           NA           NA           NA
##      Q119650.fctr Q119334.fctr Q118892.fctr Q118237.fctr Q118233.fctr
## 2328    Receiving          Yes           No           NA           NA
## 4767           NA           NA           No           No           No
##      Q118232.fctr Q118117.fctr Q117193.fctr Q117186.fctr Q116797.fctr
## 2328           NA           NA           NA           NA           NA
## 4767           Pr          Yes           NA           NA           NA
##      Q116881.fctr Q116953.fctr Q116601.fctr Q116441.fctr Q116448.fctr
## 2328           NA           NA           NA           NA           NA
## 4767           NA           NA           NA           NA           NA
##      Q116197.fctr Q115602.fctr Q115777.fctr Q115610.fctr Q115611.fctr
## 2328           NA           NA           NA           NA           NA
## 4767           NA           NA           NA           NA           NA
##      Q115899.fctr Q115390.fctr Q115195.fctr Q114961.fctr Q114748.fctr
## 2328           NA           NA           NA           NA           NA
## 4767           NA           NA           NA           NA           NA
##      Q114517.fctr Q114386.fctr Q114152.fctr Q113992.fctr Q113583.fctr
## 2328           NA           NA           NA           NA           NA
## 4767           NA           NA           NA           NA           NA
##      Q113584.fctr Q113181.fctr Q112478.fctr Q112512.fctr Q112270.fctr
## 2328           NA           NA           NA           NA           NA
## 4767           NA          Yes           NA           NA          Yes
##      Q111848.fctr Q111580.fctr Q111220.fctr Q110740.fctr Q109367.fctr
## 2328           NA           NA           NA           NA           NA
## 4767          Yes   Supportive           No          Mac          Yes
##      Q109244.fctr Q108950.fctr Q108855.fctr Q108617.fctr Q108856.fctr
## 2328           NA           NA           NA           NA           NA
## 4767           NA           NA         Yes!           No        Space
##      Q108754.fctr Q108342.fctr Q108343.fctr Q107869.fctr Q107491.fctr
## 2328           NA           NA           NA           NA           NA
## 4767           No    In-person           No           NA           NA
##      Q106993.fctr Q106997.fctr Q106272.fctr Q106388.fctr Q106389.fctr
## 2328           NA           NA           NA           NA           NA
## 4767          Yes           Yy           NA           NA           NA
##      Q106042.fctr Q105840.fctr Q105655.fctr Q104996.fctr Q103293.fctr
## 2328           NA           NA           NA           NA           NA
## 4767           NA           NA           NA           NA           NA
##      Q102906.fctr Q102674.fctr Q102687.fctr Q102289.fctr Q102089.fctr
## 2328           NA           NA           NA           NA           NA
## 4767           NA           NA           NA           NA           NA
##      Q101162.fctr Q101163.fctr Q101596.fctr Q100689.fctr Q100680.fctr
## 2328           NA           NA           NA           NA           NA
## 4767     Optimist          Dad           NA          Yes          Yes
##      Q100562.fctr Q100010.fctr Q99982.fctr Q99716.fctr Q99581.fctr
## 2328           NA          Yes      Check!          No          No
## 4767          Yes           NA          NA          NA          NA
##      Q99480.fctr Q98869.fctr Q98578.fctr Q98197.fctr Q98059.fctr
## 2328         Yes         Yes         Yes          NA          NA
## 4767          NA          NA          NA          NA          NA
##      Q98078.fctr Q96024.fctr
## 2328          NA          NA
## 4767          NA          NA
## [1] "min distance(0.9482) pair:"
##      USER_ID Party.fctr Hhold.fctr YOB.Age.fctr Gender.fctr Income.fctr
## 3363    4185          D        MKn      (30,35]           F    100-150K
## 4551    5680          D        MKn      (25,30]           F      25-50K
##      Edn.fctr Q124742.fctr Q124122.fctr Q123621.fctr Q123464.fctr
## 3363      Bcr           NA           NA           NA           NA
## 4551      Bcr           NA           NA           NA           NA
##      Q122771.fctr Q122770.fctr Q122769.fctr Q122120.fctr Q121700.fctr
## 3363           NA           NA           NA           NA           NA
## 4551           NA           NA           NA           NA           NA
##      Q121699.fctr Q121011.fctr Q120978.fctr Q120650.fctr Q120472.fctr
## 3363           NA           NA           NA           NA           NA
## 4551           NA           NA           NA           NA           NA
##      Q120379.fctr Q120194.fctr Q120014.fctr Q120012.fctr Q119851.fctr
## 3363           NA           NA           NA           NA           NA
## 4551           NA           NA           NA           NA           NA
##      Q119650.fctr Q119334.fctr Q118892.fctr Q118237.fctr Q118233.fctr
## 3363           NA           NA           NA           NA           NA
## 4551           NA           NA           NA           NA           NA
##      Q118232.fctr Q118117.fctr   Q117193.fctr Q117186.fctr Q116797.fctr
## 3363           NA           NA             NA           NA           NA
## 4551           NA           NA Standard hours           NA           NA
##      Q116881.fctr Q116953.fctr Q116601.fctr Q116441.fctr Q116448.fctr
## 3363           NA           NA           NA           NA           NA
## 4551           NA           NA           NA           NA           NA
##      Q116197.fctr Q115602.fctr Q115777.fctr Q115610.fctr Q115611.fctr
## 3363           NA           NA           NA           NA           NA
## 4551           NA           NA           NA           NA           NA
##      Q115899.fctr Q115390.fctr Q115195.fctr Q114961.fctr Q114748.fctr
## 3363           NA           NA           NA           NA           NA
## 4551           NA          Yes           NA           NA           No
##      Q114517.fctr Q114386.fctr Q114152.fctr Q113992.fctr Q113583.fctr
## 3363           NA           NA           NA           NA           NA
## 4551           NA           NA           NA           NA           NA
##      Q113584.fctr Q113181.fctr Q112478.fctr Q112512.fctr Q112270.fctr
## 3363           NA           NA           NA           NA           NA
## 4551           NA           NA           NA           NA           NA
##      Q111848.fctr Q111580.fctr Q111220.fctr Q110740.fctr Q109367.fctr
## 3363           NA           NA           NA           NA           NA
## 4551           NA           NA           NA           NA           NA
##      Q109244.fctr Q108950.fctr Q108855.fctr Q108617.fctr Q108856.fctr
## 3363          Yes           NA           NA           NA           NA
## 4551          Yes     Cautious       Umm...           NA        Space
##      Q108754.fctr Q108342.fctr Q108343.fctr Q107869.fctr Q107491.fctr
## 3363           NA           NA           NA           NA           NA
## 4551           NA           NA           NA           NA           NA
##      Q106993.fctr Q106997.fctr Q106272.fctr Q106388.fctr Q106389.fctr
## 3363           NA           NA           NA           NA           NA
## 4551           NA           NA           NA           NA           NA
##      Q106042.fctr Q105840.fctr Q105655.fctr Q104996.fctr Q103293.fctr
## 3363           NA           NA           NA           NA           NA
## 4551           NA           NA          Yes           NA           NA
##      Q102906.fctr Q102674.fctr Q102687.fctr Q102289.fctr Q102089.fctr
## 3363           NA           NA           NA           NA           NA
## 4551           NA          Yes          Yes           NA           NA
##      Q101162.fctr Q101163.fctr Q101596.fctr Q100689.fctr Q100680.fctr
## 3363           NA           NA           NA           NA           NA
## 4551           NA           NA           NA           NA           NA
##      Q100562.fctr Q100010.fctr Q99982.fctr Q99716.fctr Q99581.fctr
## 3363           NA           NA          NA          NA          NA
## 4551           NA           NA          NA          NA          NA
##      Q99480.fctr Q98869.fctr Q98578.fctr Q98197.fctr Q98059.fctr
## 3363          NA          NA          NA          NA          NA
## 4551          NA          NA          NA          NA          NA
##      Q98078.fctr Q96024.fctr
## 3363          NA          NA
## 4551          NA          NA
## [1] "Category: MKy"
## [1] "max distance(0.9786) pair:"
##      USER_ID Party.fctr Hhold.fctr YOB.Age.fctr Gender.fctr Income.fctr
## 3492    4349          R        MKy      (65,90]           F    100-150K
## 4106    5121          R        MKy           NA           N           N
##      Edn.fctr Q124742.fctr Q124122.fctr Q123621.fctr Q123464.fctr
## 3492        N           NA          Yes           NA           NA
## 4106      Msr           NA           NA          Yes           NA
##      Q122771.fctr Q122770.fctr Q122769.fctr Q122120.fctr Q121700.fctr
## 3492           Pt          Yes           NA           NA           NA
## 4106           Pt           NA           NA           No           NA
##      Q121699.fctr Q121011.fctr Q120978.fctr Q120650.fctr Q120472.fctr
## 3492           NA           NA           NA           NA           NA
## 4106           NA           NA          Yes          Yes      Science
##      Q120379.fctr Q120194.fctr Q120014.fctr Q120012.fctr Q119851.fctr
## 3492           NA           NA           NA          Yes           NA
## 4106          Yes  Study first           No          Yes          Yes
##      Q119650.fctr Q119334.fctr Q118892.fctr Q118237.fctr Q118233.fctr
## 3492           NA           NA           NA           NA           NA
## 4106       Giving           NA          Yes           NA           NA
##      Q118232.fctr Q118117.fctr   Q117193.fctr Q117186.fctr Q116797.fctr
## 3492           NA           NA             NA   Hot headed          Yes
## 4106           NA           No Standard hours           NA           NA
##      Q116881.fctr Q116953.fctr Q116601.fctr Q116441.fctr Q116448.fctr
## 3492           NA           NA          Yes           No           NA
## 4106           NA          Yes          Yes          Yes           No
##      Q116197.fctr Q115602.fctr Q115777.fctr Q115610.fctr Q115611.fctr
## 3492         A.M.           NA           NA          Yes           NA
## 4106         A.M.          Yes           NA           NA           NA
##      Q115899.fctr Q115390.fctr Q115195.fctr Q114961.fctr Q114748.fctr
## 3492           Me           NA           NA          Yes           NA
## 4106           NA           NA           No           NA          Yes
##      Q114517.fctr Q114386.fctr Q114152.fctr Q113992.fctr Q113583.fctr
## 3492          Yes           NA           NA           NA         Talk
## 4106           NA           NA           NA           NA        Tunes
##      Q113584.fctr Q113181.fctr Q112478.fctr Q112512.fctr Q112270.fctr
## 3492           NA           NA          Yes          Yes           NA
## 4106       People           NA          Yes           NA           NA
##      Q111848.fctr Q111580.fctr Q111220.fctr Q110740.fctr Q109367.fctr
## 3492           NA           NA           NA          Mac           NA
## 4106          Yes           NA           NA           NA           No
##      Q109244.fctr Q108950.fctr Q108855.fctr Q108617.fctr Q108856.fctr
## 3492           NA           NA       Umm...           NA        Space
## 4106           NA           NA           NA           NA           NA
##      Q108754.fctr Q108342.fctr Q108343.fctr Q107869.fctr Q107491.fctr
## 3492           NA       Online           NA           NA          Yes
## 4106           NA           NA           NA           NA           NA
##      Q106993.fctr Q106997.fctr Q106272.fctr Q106388.fctr Q106389.fctr
## 3492           NA           NA           NA           NA           NA
## 4106           No           NA          Yes           No           NA
##      Q106042.fctr Q105840.fctr Q105655.fctr Q104996.fctr Q103293.fctr
## 3492          Yes           NA           NA           NA           NA
## 4106           NA           No          Yes          Yes           NA
##      Q102906.fctr Q102674.fctr Q102687.fctr Q102289.fctr Q102089.fctr
## 3492           NA           NA           NA           NA          Own
## 4106           NA           NA          Yes           NA           NA
##      Q101162.fctr Q101163.fctr Q101596.fctr Q100689.fctr Q100680.fctr
## 3492    Pessimist           NA           NA           NA           NA
## 4106     Optimist           NA           No          Yes          Yes
##      Q100562.fctr Q100010.fctr Q99982.fctr Q99716.fctr Q99581.fctr
## 3492           NA           NA          NA          NA          NA
## 4106           NA           No      Check!          No          No
##      Q99480.fctr Q98869.fctr Q98578.fctr Q98197.fctr Q98059.fctr
## 3492          NA          NA          NA          NA          NA
## 4106         Yes          NA         Yes          NA         Yes
##      Q98078.fctr Q96024.fctr
## 3492          NA          NA
## 4106          NA         Yes
## [1] "min distance(0.9489) pair:"
##      USER_ID Party.fctr Hhold.fctr YOB.Age.fctr Gender.fctr Income.fctr
## 4998    6245          R        MKy      (15,20]           M      50-75K
## 6032    2409       <NA>        MKy      (35,40]           M        <25K
##      Edn.fctr Q124742.fctr Q124122.fctr Q123621.fctr Q123464.fctr
## 4998        N           NA           NA           NA           NA
## 6032        N           NA           NA           NA           NA
##      Q122771.fctr Q122770.fctr Q122769.fctr Q122120.fctr Q121700.fctr
## 4998           NA           NA           NA           NA           NA
## 6032           NA           NA           NA           NA           NA
##      Q121699.fctr Q121011.fctr Q120978.fctr Q120650.fctr Q120472.fctr
## 4998           NA           NA           NA           NA      Science
## 6032           NA           NA           NA           NA          Art
##      Q120379.fctr Q120194.fctr Q120014.fctr Q120012.fctr Q119851.fctr
## 4998          Yes    Try first           NA          Yes          Yes
## 6032           NA           NA           NA           NA           NA
##      Q119650.fctr Q119334.fctr Q118892.fctr Q118237.fctr Q118233.fctr
## 4998       Giving           NA           NA           NA           NA
## 6032           NA           NA           No           No           No
##      Q118232.fctr Q118117.fctr Q117193.fctr Q117186.fctr Q116797.fctr
## 4998           Pr           NA           NA  Cool headed           NA
## 6032           Id          Yes           NA           NA           NA
##      Q116881.fctr Q116953.fctr Q116601.fctr Q116441.fctr Q116448.fctr
## 4998        Right           No           NA           NA           NA
## 6032           NA           NA           NA           NA           NA
##      Q116197.fctr Q115602.fctr Q115777.fctr Q115610.fctr Q115611.fctr
## 4998           NA           NA           NA           NA           NA
## 6032           NA           NA           NA           NA           NA
##      Q115899.fctr Q115390.fctr Q115195.fctr Q114961.fctr Q114748.fctr
## 4998           NA           NA           NA           NA           NA
## 6032           NA           NA           NA           NA           NA
##      Q114517.fctr Q114386.fctr Q114152.fctr Q113992.fctr Q113583.fctr
## 4998           NA           NA           NA           NA           NA
## 6032           NA           NA           NA           NA           NA
##      Q113584.fctr Q113181.fctr Q112478.fctr Q112512.fctr Q112270.fctr
## 4998           NA           NA           NA           NA           NA
## 6032           NA           NA           NA           NA           NA
##      Q111848.fctr Q111580.fctr Q111220.fctr Q110740.fctr Q109367.fctr
## 4998           NA   Supportive           NA           NA           NA
## 6032           NA           NA           NA           NA           NA
##      Q109244.fctr Q108950.fctr Q108855.fctr Q108617.fctr Q108856.fctr
## 4998          Yes           NA           NA          Yes           NA
## 6032           No           NA           NA           NA           NA
##      Q108754.fctr Q108342.fctr Q108343.fctr Q107869.fctr Q107491.fctr
## 4998          Yes           NA           NA           NA           NA
## 6032           NA           NA           NA           No           No
##      Q106993.fctr Q106997.fctr Q106272.fctr Q106388.fctr Q106389.fctr
## 4998           NA           NA           NA           NA           NA
## 6032           NA           NA           NA           NA           NA
##      Q106042.fctr Q105840.fctr Q105655.fctr Q104996.fctr Q103293.fctr
## 4998           NA           NA           NA           NA           NA
## 6032           NA           NA           NA           NA           NA
##      Q102906.fctr Q102674.fctr Q102687.fctr Q102289.fctr Q102089.fctr
## 4998           NA           NA           NA           NA           NA
## 6032           NA           NA           NA           NA           NA
##      Q101162.fctr Q101163.fctr Q101596.fctr Q100689.fctr Q100680.fctr
## 4998           NA           NA           NA           NA           NA
## 6032           NA           NA           NA           NA          Yes
##      Q100562.fctr Q100010.fctr Q99982.fctr Q99716.fctr Q99581.fctr
## 4998           NA           NA          NA          NA          NA
## 6032          Yes          Yes        Nope          No         Yes
##      Q99480.fctr Q98869.fctr Q98578.fctr Q98197.fctr Q98059.fctr
## 4998          NA          NA          NA          NA          NA
## 6032          NA          NA          NA          NA          NA
##      Q98078.fctr Q96024.fctr
## 4998          NA          NA
## 6032          NA          NA
## [1] "Category: PKn"
## [1] "max distance(0.9742) pair:"
##      USER_ID Party.fctr Hhold.fctr YOB.Age.fctr Gender.fctr Income.fctr
## 404      502          D        PKn      (40,50]           F     75-100K
## 6145    2930       <NA>        PKn      (50,65]           F      25-50K
##      Edn.fctr Q124742.fctr Q124122.fctr Q123621.fctr Q123464.fctr
## 404       PhD           NA          Yes           NA           NA
## 6145        N           NA           NA          Yes           No
##      Q122771.fctr Q122770.fctr Q122769.fctr Q122120.fctr Q121700.fctr
## 404            NA           NA           NA           NA           No
## 6145           Pc           No          Yes           No           No
##      Q121699.fctr Q121011.fctr Q120978.fctr Q120650.fctr Q120472.fctr
## 404           Yes          Yes          Yes           NA           NA
## 6145          Yes           No          Yes          Yes          Art
##      Q120379.fctr Q120194.fctr Q120014.fctr Q120012.fctr Q119851.fctr
## 404            NA           NA           NA           NA           NA
## 6145          Yes    Try first          Yes          Yes          Yes
##      Q119650.fctr Q119334.fctr Q118892.fctr Q118237.fctr Q118233.fctr
## 404            NA           NA           NA           NA           NA
## 6145       Giving           No           NA           NA           NA
##      Q118232.fctr Q118117.fctr Q117193.fctr Q117186.fctr Q116797.fctr
## 404            NA           NA           NA           NA           NA
## 6145           NA           NA           NA           NA           NA
##      Q116881.fctr Q116953.fctr Q116601.fctr Q116441.fctr Q116448.fctr
## 404            NA          Yes          Yes          Yes           NA
## 6145           NA           NA           NA           NA           NA
##      Q116197.fctr Q115602.fctr Q115777.fctr Q115610.fctr Q115611.fctr
## 404          P.M.           NA           NA           NA           No
## 6145           NA           NA           NA           NA           NA
##      Q115899.fctr Q115390.fctr Q115195.fctr Q114961.fctr Q114748.fctr
## 404            NA           NA           NA           NA           NA
## 6145           NA           NA           NA           NA           NA
##      Q114517.fctr Q114386.fctr Q114152.fctr Q113992.fctr Q113583.fctr
## 404            NA           NA           NA           NA           NA
## 6145           NA           NA           NA           NA           NA
##      Q113584.fctr Q113181.fctr Q112478.fctr Q112512.fctr Q112270.fctr
## 404            NA           NA           NA           NA           NA
## 6145           NA           NA           NA           NA           NA
##      Q111848.fctr Q111580.fctr Q111220.fctr Q110740.fctr Q109367.fctr
## 404            NA           NA           NA           NA           NA
## 6145           NA           NA           NA           NA           NA
##      Q109244.fctr Q108950.fctr Q108855.fctr Q108617.fctr Q108856.fctr
## 404            NA           NA           NA           NA           NA
## 6145           NA           NA           NA           NA           NA
##      Q108754.fctr Q108342.fctr Q108343.fctr Q107869.fctr Q107491.fctr
## 404            NA           NA           NA           No          Yes
## 6145           NA           NA           NA           NA           NA
##      Q106993.fctr Q106997.fctr Q106272.fctr Q106388.fctr Q106389.fctr
## 404            NA           NA           NA           NA           NA
## 6145           NA           NA           NA           NA           NA
##      Q106042.fctr Q105840.fctr Q105655.fctr Q104996.fctr Q103293.fctr
## 404            NA          Yes          Yes          Yes          Yes
## 6145           NA           NA           NA           NA           NA
##      Q102906.fctr Q102674.fctr Q102687.fctr Q102289.fctr Q102089.fctr
## 404            No           No           No           No         Rent
## 6145           NA           NA           NA           NA           NA
##      Q101162.fctr Q101163.fctr Q101596.fctr Q100689.fctr Q100680.fctr
## 404      Optimist          Mom           No          Yes           NA
## 6145           NA           NA           NA           NA           NA
##      Q100562.fctr Q100010.fctr Q99982.fctr Q99716.fctr Q99581.fctr
## 404           Yes           NA      Check!          No         Yes
## 6145           NA           NA          NA          NA          NA
##      Q99480.fctr Q98869.fctr Q98578.fctr Q98197.fctr Q98059.fctr
## 404          Yes          NA          NA         Yes         Yes
## 6145          NA          NA          NA          NA          NA
##      Q98078.fctr Q96024.fctr
## 404          Yes          No
## 6145          NA          NA
## [1] "min distance(0.9462) pair:"
##      USER_ID Party.fctr Hhold.fctr YOB.Age.fctr Gender.fctr Income.fctr
## 848     1046          D        PKn      (50,65]           M    100-150K
## 3463    4312          D        PKn      (20,25]           F      25-50K
##      Edn.fctr Q124742.fctr Q124122.fctr Q123621.fctr Q123464.fctr
## 848         N           NA           NA           NA           NA
## 3463      HSD           NA           NA           NA           NA
##      Q122771.fctr Q122770.fctr Q122769.fctr Q122120.fctr Q121700.fctr
## 848            NA           NA           NA           NA           NA
## 3463           NA           NA           NA           NA           NA
##      Q121699.fctr Q121011.fctr Q120978.fctr Q120650.fctr Q120472.fctr
## 848            NA           NA           NA           NA           NA
## 3463           NA           NA           NA           NA           NA
##      Q120379.fctr Q120194.fctr Q120014.fctr Q120012.fctr Q119851.fctr
## 848            NA           NA           NA           NA           NA
## 3463           NA           NA           NA           NA           NA
##      Q119650.fctr Q119334.fctr Q118892.fctr Q118237.fctr Q118233.fctr
## 848            NA           NA           NA           NA           NA
## 3463           NA           NA           NA           NA           NA
##      Q118232.fctr Q118117.fctr Q117193.fctr Q117186.fctr Q116797.fctr
## 848            NA           NA           NA           NA           NA
## 3463           NA           NA           NA           NA           NA
##      Q116881.fctr Q116953.fctr Q116601.fctr Q116441.fctr Q116448.fctr
## 848            NA           NA           NA           NA           NA
## 3463           NA           NA           NA           NA           NA
##      Q116197.fctr Q115602.fctr Q115777.fctr Q115610.fctr Q115611.fctr
## 848            NA           NA           NA           NA           NA
## 3463           NA           NA           NA           NA           NA
##      Q115899.fctr Q115390.fctr Q115195.fctr Q114961.fctr Q114748.fctr
## 848            NA           NA           NA           NA           NA
## 3463           NA           NA           NA           NA           NA
##      Q114517.fctr Q114386.fctr Q114152.fctr Q113992.fctr Q113583.fctr
## 848            NA           NA           NA           NA           NA
## 3463           NA           NA           NA           NA           NA
##      Q113584.fctr Q113181.fctr Q112478.fctr Q112512.fctr Q112270.fctr
## 848            NA           NA           NA           NA           NA
## 3463           NA           NA           NA           NA           NA
##      Q111848.fctr Q111580.fctr Q111220.fctr Q110740.fctr Q109367.fctr
## 848            NA           NA           NA           NA          Yes
## 3463           NA           NA          Yes          Mac          Yes
##      Q109244.fctr  Q108950.fctr Q108855.fctr Q108617.fctr Q108856.fctr
## 848           Yes Risk-friendly         Yes!           No        Space
## 3463          Yes Risk-friendly         Yes!           No        Space
##      Q108754.fctr Q108342.fctr Q108343.fctr Q107869.fctr Q107491.fctr
## 848            No    In-person          Yes           NA           NA
## 3463          Yes           NA           NA           NA           NA
##      Q106993.fctr Q106997.fctr Q106272.fctr Q106388.fctr Q106389.fctr
## 848            NA           NA           NA           NA           NA
## 3463           NA           NA           NA           NA           NA
##      Q106042.fctr Q105840.fctr Q105655.fctr Q104996.fctr Q103293.fctr
## 848            NA           NA           NA           NA           NA
## 3463           NA           NA           NA           NA           NA
##      Q102906.fctr Q102674.fctr Q102687.fctr Q102289.fctr Q102089.fctr
## 848            NA           NA           NA           NA           NA
## 3463           NA           NA           NA           NA           NA
##      Q101162.fctr Q101163.fctr Q101596.fctr Q100689.fctr Q100680.fctr
## 848            NA           NA           NA           NA           NA
## 3463           NA           NA           NA           NA           NA
##      Q100562.fctr Q100010.fctr Q99982.fctr Q99716.fctr Q99581.fctr
## 848            NA           NA          NA          NA          NA
## 3463           NA           NA          NA          NA          NA
##      Q99480.fctr Q98869.fctr Q98578.fctr Q98197.fctr Q98059.fctr
## 848           NA          NA          NA          NA          NA
## 3463          NA          NA          NA          NA          NA
##      Q98078.fctr Q96024.fctr
## 848           NA          No
## 3463          NA          NA
## [1] "Category: PKy"
## [1] "max distance(0.9739) pair:"
##      USER_ID Party.fctr Hhold.fctr YOB.Age.fctr Gender.fctr Income.fctr
## 1561    1933          R        PKy      (50,65]           F       >150K
## 4384    5471          R        PKy      (30,35]           F    100-150K
##      Edn.fctr Q124742.fctr Q124122.fctr Q123621.fctr Q123464.fctr
## 1561      Ast           NA           NA           NA           NA
## 4384      HSD          Yes          Yes          Yes           No
##      Q122771.fctr Q122770.fctr Q122769.fctr Q122120.fctr Q121700.fctr
## 1561           NA           NA           NA           NA           NA
## 4384           Pc           No          Yes           No          Yes
##      Q121699.fctr Q121011.fctr Q120978.fctr Q120650.fctr Q120472.fctr
## 1561           NA           NA           NA           NA           NA
## 4384          Yes           No          Yes          Yes          Art
##      Q120379.fctr Q120194.fctr Q120014.fctr Q120012.fctr Q119851.fctr
## 1561           NA           NA           NA           NA           NA
## 4384           No  Study first          Yes          Yes          Yes
##      Q119650.fctr Q119334.fctr Q118892.fctr Q118237.fctr Q118233.fctr
## 1561           NA           NA           NA          Yes          Yes
## 4384       Giving          Yes           No           No          Yes
##      Q118232.fctr Q118117.fctr   Q117193.fctr Q117186.fctr Q116797.fctr
## 1561           Pr           No             NA  Cool headed           No
## 4384           Id           No Standard hours   Hot headed           No
##      Q116881.fctr Q116953.fctr Q116601.fctr Q116441.fctr Q116448.fctr
## 1561        Happy           No          Yes           No           No
## 4384        Right           No           NA           NA           NA
##      Q116197.fctr Q115602.fctr Q115777.fctr Q115610.fctr Q115611.fctr
## 1561         A.M.          Yes        Start          Yes           NA
## 4384           NA           NA           NA           NA           NA
##      Q115899.fctr Q115390.fctr Q115195.fctr Q114961.fctr Q114748.fctr
## 1561           Cs          Yes           No           No          Yes
## 4384           NA           NA           NA           NA           NA
##      Q114517.fctr Q114386.fctr Q114152.fctr Q113992.fctr Q113583.fctr
## 1561          Yes   Mysterious           No           No           NA
## 4384           NA           NA           NA           NA           NA
##      Q113584.fctr Q113181.fctr Q112478.fctr Q112512.fctr Q112270.fctr
## 1561           NA           NA           NA           NA           NA
## 4384           NA           NA           NA           NA           NA
##      Q111848.fctr Q111580.fctr Q111220.fctr Q110740.fctr Q109367.fctr
## 1561           NA           NA           NA           NA           NA
## 4384           NA           NA           NA           NA           NA
##      Q109244.fctr Q108950.fctr Q108855.fctr Q108617.fctr Q108856.fctr
## 1561           NA           NA           NA           NA           NA
## 4384           NA           NA           NA           NA           NA
##      Q108754.fctr Q108342.fctr Q108343.fctr Q107869.fctr Q107491.fctr
## 1561           NA           NA           NA           NA           NA
## 4384           NA           NA           NA           NA           NA
##      Q106993.fctr Q106997.fctr Q106272.fctr Q106388.fctr Q106389.fctr
## 1561           NA           NA          Yes           No           No
## 4384           NA           NA           NA           NA           NA
##      Q106042.fctr Q105840.fctr Q105655.fctr Q104996.fctr Q103293.fctr
## 1561           No          Yes           No           NA           NA
## 4384           NA           NA           NA           NA           NA
##      Q102906.fctr Q102674.fctr Q102687.fctr Q102289.fctr Q102089.fctr
## 1561           NA           NA           NA           NA           NA
## 4384           NA          Yes           No          Yes         Rent
##      Q101162.fctr Q101163.fctr Q101596.fctr Q100689.fctr Q100680.fctr
## 1561           NA           NA           NA          Yes          Yes
## 4384     Optimist          Dad          Yes          Yes           NA
##      Q100562.fctr Q100010.fctr Q99982.fctr Q99716.fctr Q99581.fctr
## 1561           No          Yes      Check!          No          No
## 4384           NA           NA          NA          NA          NA
##      Q99480.fctr Q98869.fctr Q98578.fctr Q98197.fctr Q98059.fctr
## 1561         Yes         Yes          No          No         Yes
## 4384          NA          NA          NA          NA          NA
##      Q98078.fctr Q96024.fctr
## 1561         Yes         Yes
## 4384          NA          No
## [1] "min distance(0.9528) pair:"
##      USER_ID Party.fctr Hhold.fctr YOB.Age.fctr Gender.fctr Income.fctr
## 3332    4150          D        PKy      (25,30]           M      50-75K
## 6815    6244       <NA>        PKy      (40,50]           M           N
##      Edn.fctr Q124742.fctr Q124122.fctr Q123621.fctr Q123464.fctr
## 3332      HSD           NA           NA           NA           NA
## 6815      Ast           NA           NA           NA           NA
##      Q122771.fctr Q122770.fctr Q122769.fctr Q122120.fctr Q121700.fctr
## 3332           Pc           No          Yes           NA           No
## 6815           NA           NA           NA           NA           NA
##      Q121699.fctr Q121011.fctr Q120978.fctr Q120650.fctr Q120472.fctr
## 3332           No           No           No           NA      Science
## 6815           NA           NA           NA           NA           NA
##      Q120379.fctr Q120194.fctr Q120014.fctr Q120012.fctr Q119851.fctr
## 3332           No           NA           NA           No           No
## 6815           NA           NA           NA           NA           NA
##      Q119650.fctr Q119334.fctr Q118892.fctr Q118237.fctr Q118233.fctr
## 3332           NA           NA           NA           NA           NA
## 6815           NA           NA           NA           NA           NA
##      Q118232.fctr Q118117.fctr Q117193.fctr Q117186.fctr Q116797.fctr
## 3332           NA           NA    Odd hours  Cool headed           NA
## 6815           NA           NA           NA           NA           NA
##      Q116881.fctr Q116953.fctr Q116601.fctr Q116441.fctr Q116448.fctr
## 3332        Right           NA           NA           NA           NA
## 6815           NA           NA           NA           NA           NA
##      Q116197.fctr Q115602.fctr Q115777.fctr Q115610.fctr Q115611.fctr
## 3332           NA           NA           NA           NA           NA
## 6815           NA           NA           NA           NA           NA
##      Q115899.fctr Q115390.fctr Q115195.fctr Q114961.fctr Q114748.fctr
## 3332           NA           NA           NA           NA           No
## 6815           NA           NA           NA           NA           NA
##      Q114517.fctr Q114386.fctr Q114152.fctr Q113992.fctr Q113583.fctr
## 3332           No           NA           NA           No           NA
## 6815           NA           NA           NA           NA           NA
##      Q113584.fctr Q113181.fctr Q112478.fctr Q112512.fctr Q112270.fctr
## 3332           NA           No          Yes          Yes           NA
## 6815           NA           NA           NA           NA           NA
##      Q111848.fctr Q111580.fctr Q111220.fctr Q110740.fctr Q109367.fctr
## 3332           NA           NA           NA           NA           No
## 6815           NA           NA           NA           PC          Yes
##      Q109244.fctr  Q108950.fctr Q108855.fctr Q108617.fctr Q108856.fctr
## 3332           No      Cautious           NA           No           NA
## 6815          Yes Risk-friendly       Umm...           No        Space
##      Q108754.fctr Q108342.fctr Q108343.fctr Q107869.fctr Q107491.fctr
## 3332           NA    In-person          Yes           NA           NA
## 6815           No           NA           NA           NA           NA
##      Q106993.fctr Q106997.fctr Q106272.fctr Q106388.fctr Q106389.fctr
## 3332          Yes           Gr           NA           NA           NA
## 6815           NA           NA           NA           NA           NA
##      Q106042.fctr Q105840.fctr Q105655.fctr Q104996.fctr Q103293.fctr
## 3332           NA           NA           NA           No           NA
## 6815           NA           NA           NA           NA           NA
##      Q102906.fctr Q102674.fctr Q102687.fctr Q102289.fctr Q102089.fctr
## 3332          Yes           NA           No           NA           NA
## 6815           NA           NA           NA           NA           NA
##      Q101162.fctr Q101163.fctr Q101596.fctr Q100689.fctr Q100680.fctr
## 3332    Pessimist           NA           NA          Yes           NA
## 6815           NA           NA           NA           NA           NA
##      Q100562.fctr Q100010.fctr Q99982.fctr Q99716.fctr Q99581.fctr
## 3332           No           NA          NA          NA          NA
## 6815           NA           NA          NA          NA          NA
##      Q99480.fctr Q98869.fctr Q98578.fctr Q98197.fctr Q98059.fctr
## 3332          NA          NA          NA          NA          NA
## 6815          NA          NA          NA          NA          NA
##      Q98078.fctr Q96024.fctr
## 3332          NA          NA
## 6815          NA          NA
## [1] "Category: SKn"
## [1] "max distance(0.9784) pair:"
##      USER_ID Party.fctr Hhold.fctr YOB.Age.fctr Gender.fctr Income.fctr
## 5083    6347          R        SKn           NA           N       >150K
## 6482    4629       <NA>        SKn      (30,35]           N      50-75K
##      Edn.fctr Q124742.fctr Q124122.fctr Q123621.fctr Q123464.fctr
## 5083      PhD           NA           NA           NA           NA
## 6482        N           NA           NA           NA           NA
##      Q122771.fctr Q122770.fctr Q122769.fctr Q122120.fctr Q121700.fctr
## 5083           Pt           NA           NA           NA           NA
## 6482           NA           NA           NA           NA           No
##      Q121699.fctr Q121011.fctr Q120978.fctr Q120650.fctr Q120472.fctr
## 5083           NA           NA           NA           NA           NA
## 6482          Yes          Yes          Yes          Yes          Art
##      Q120379.fctr Q120194.fctr Q120014.fctr Q120012.fctr Q119851.fctr
## 5083          Yes           NA           NA           NA           NA
## 6482           No  Study first           No           No           No
##      Q119650.fctr Q119334.fctr Q118892.fctr Q118237.fctr Q118233.fctr
## 5083           NA           NA           NA           NA           NA
## 6482    Receiving          Yes           NA           NA           NA
##      Q118232.fctr Q118117.fctr Q117193.fctr Q117186.fctr Q116797.fctr
## 5083           NA           NA           NA           NA           No
## 6482           NA           NA           NA           NA           NA
##      Q116881.fctr Q116953.fctr Q116601.fctr Q116441.fctr Q116448.fctr
## 5083           NA           NA           NA           NA           NA
## 6482           NA           NA           NA           NA           NA
##      Q116197.fctr Q115602.fctr Q115777.fctr Q115610.fctr Q115611.fctr
## 5083           NA           NA           NA           NA           NA
## 6482           NA           NA           NA           NA           NA
##      Q115899.fctr Q115390.fctr Q115195.fctr Q114961.fctr Q114748.fctr
## 5083           NA           NA           NA           NA          Yes
## 6482           NA           NA           NA           NA           NA
##      Q114517.fctr Q114386.fctr Q114152.fctr Q113992.fctr Q113583.fctr
## 5083           NA           NA           NA           NA           NA
## 6482           NA           NA           NA           NA           NA
##      Q113584.fctr Q113181.fctr Q112478.fctr Q112512.fctr Q112270.fctr
## 5083           NA           NA           NA           NA           NA
## 6482           NA           NA           NA           NA           NA
##      Q111848.fctr Q111580.fctr Q111220.fctr Q110740.fctr Q109367.fctr
## 5083           NA           NA           NA           NA           NA
## 6482           NA           NA           NA           NA           NA
##      Q109244.fctr Q108950.fctr Q108855.fctr Q108617.fctr Q108856.fctr
## 5083           NA           NA           NA           NA           NA
## 6482           NA           NA           NA           NA           NA
##      Q108754.fctr Q108342.fctr Q108343.fctr Q107869.fctr Q107491.fctr
## 5083           NA           NA           NA           NA           NA
## 6482           NA           NA           NA          Yes          Yes
##      Q106993.fctr Q106997.fctr Q106272.fctr Q106388.fctr Q106389.fctr
## 5083           NA           NA           NA           NA           NA
## 6482           NA           NA          Yes           No           No
##      Q106042.fctr Q105840.fctr Q105655.fctr Q104996.fctr Q103293.fctr
## 5083           NA           NA           NA           NA           NA
## 6482          Yes           No          Yes           No           No
##      Q102906.fctr Q102674.fctr Q102687.fctr Q102289.fctr Q102089.fctr
## 5083           NA           NA           NA           NA           NA
## 6482           No           No          Yes          Yes         Rent
##      Q101162.fctr Q101163.fctr Q101596.fctr Q100689.fctr Q100680.fctr
## 5083           NA           NA           NA           NA           NA
## 6482     Optimist          Mom          Yes           NA           NA
##      Q100562.fctr Q100010.fctr Q99982.fctr Q99716.fctr Q99581.fctr
## 5083           NA           NA          NA          NA          NA
## 6482           NA          Yes      Check!          No          No
##      Q99480.fctr Q98869.fctr Q98578.fctr Q98197.fctr Q98059.fctr
## 5083          NA          NA          NA          NA          NA
## 6482         Yes         Yes          No          No  Only-child
##      Q98078.fctr Q96024.fctr
## 5083          NA          NA
## 6482         Yes          NA
## [1] "min distance(0.9355) pair:"
##      USER_ID Party.fctr Hhold.fctr YOB.Age.fctr Gender.fctr Income.fctr
## 2712    3375          D        SKn      (25,30]           M        <25K
## 4692    5856          R        SKn      (25,30]           M           N
##      Edn.fctr Q124742.fctr Q124122.fctr Q123621.fctr Q123464.fctr
## 2712      HSD           NA           NA           NA           NA
## 4692      K12           NA           NA           NA           NA
##      Q122771.fctr Q122770.fctr Q122769.fctr Q122120.fctr Q121700.fctr
## 2712           NA           NA           NA           NA           NA
## 4692           NA           NA           NA           NA           NA
##      Q121699.fctr Q121011.fctr Q120978.fctr Q120650.fctr Q120472.fctr
## 2712           NA           NA           NA           NA           NA
## 4692           NA           NA           NA           NA           NA
##      Q120379.fctr Q120194.fctr Q120014.fctr Q120012.fctr Q119851.fctr
## 2712           NA           NA           NA           NA           NA
## 4692           NA           NA           NA           NA           NA
##      Q119650.fctr Q119334.fctr Q118892.fctr Q118237.fctr Q118233.fctr
## 2712           NA           NA           NA           NA           NA
## 4692           NA           NA           NA           NA           NA
##      Q118232.fctr Q118117.fctr Q117193.fctr Q117186.fctr Q116797.fctr
## 2712           NA           NA           NA           NA           NA
## 4692           NA           NA           NA           NA           NA
##      Q116881.fctr Q116953.fctr Q116601.fctr Q116441.fctr Q116448.fctr
## 2712           NA           NA           NA           NA           NA
## 4692           NA           NA           NA           NA           NA
##      Q116197.fctr Q115602.fctr Q115777.fctr Q115610.fctr Q115611.fctr
## 2712           NA           NA           NA           NA           NA
## 4692           NA           NA           NA           NA           NA
##      Q115899.fctr Q115390.fctr Q115195.fctr Q114961.fctr Q114748.fctr
## 2712           NA           NA           NA           NA           NA
## 4692           NA           NA           NA           NA           NA
##      Q114517.fctr Q114386.fctr Q114152.fctr Q113992.fctr Q113583.fctr
## 2712           NA           NA           NA           NA           NA
## 4692           NA           NA           NA           NA           NA
##      Q113584.fctr Q113181.fctr Q112478.fctr Q112512.fctr Q112270.fctr
## 2712           NA           NA           NA           NA           NA
## 4692           NA           NA           NA           NA           NA
##      Q111848.fctr Q111580.fctr Q111220.fctr Q110740.fctr Q109367.fctr
## 2712           NA           NA           NA           NA           No
## 4692           NA           NA           No           PC          Yes
##      Q109244.fctr  Q108950.fctr Q108855.fctr Q108617.fctr Q108856.fctr
## 2712          Yes      Cautious           NA           NA           NA
## 4692          Yes Risk-friendly       Umm...           NA        Space
##      Q108754.fctr Q108342.fctr Q108343.fctr Q107869.fctr Q107491.fctr
## 2712           No           NA           NA           NA           NA
## 4692          Yes           NA           NA           NA           NA
##      Q106993.fctr Q106997.fctr Q106272.fctr Q106388.fctr Q106389.fctr
## 2712           NA           NA           NA           NA           NA
## 4692           NA           NA           NA           NA           NA
##      Q106042.fctr Q105840.fctr Q105655.fctr Q104996.fctr Q103293.fctr
## 2712           NA           NA           NA           NA           NA
## 4692           NA           NA           NA           NA           NA
##      Q102906.fctr Q102674.fctr Q102687.fctr Q102289.fctr Q102089.fctr
## 2712           NA           NA           NA           NA           NA
## 4692           NA           NA           NA           NA           NA
##      Q101162.fctr Q101163.fctr Q101596.fctr Q100689.fctr Q100680.fctr
## 2712           NA           NA           NA           NA           NA
## 4692           NA           NA           NA           NA           NA
##      Q100562.fctr Q100010.fctr Q99982.fctr Q99716.fctr Q99581.fctr
## 2712           NA           NA          NA          NA          NA
## 4692           NA           NA          NA          NA          NA
##      Q99480.fctr Q98869.fctr Q98578.fctr Q98197.fctr Q98059.fctr
## 2712          NA          NA          NA          NA          NA
## 4692          NA          NA          NA          NA          NA
##      Q98078.fctr Q96024.fctr
## 2712          NA          NA
## 4692          NA          No
## [1] "Category: SKy"
## [1] "max distance(0.9771) pair:"
##      USER_ID Party.fctr Hhold.fctr YOB.Age.fctr Gender.fctr Income.fctr
## 3012    3748          R        SKy      (50,65]           F     75-100K
## 5348    6679          D        SKy      (40,50]           N     75-100K
##      Edn.fctr Q124742.fctr Q124122.fctr Q123621.fctr Q123464.fctr
## 3012        N           NA           NA           NA           NA
## 5348      PhD          Yes           No          Yes           No
##      Q122771.fctr Q122770.fctr Q122769.fctr Q122120.fctr Q121700.fctr
## 3012           NA           NA           NA           NA           NA
## 5348           Pc           No          Yes           No           No
##      Q121699.fctr Q121011.fctr Q120978.fctr Q120650.fctr Q120472.fctr
## 3012           NA           NA           NA          Yes          Art
## 5348          Yes           No          Yes          Yes          Art
##      Q120379.fctr Q120194.fctr Q120014.fctr Q120012.fctr Q119851.fctr
## 3012           No  Study first          Yes           No          Yes
## 5348           No           NA           NA          Yes           NA
##      Q119650.fctr Q119334.fctr Q118892.fctr Q118237.fctr Q118233.fctr
## 3012       Giving           No          Yes           No          Yes
## 5348           NA           NA          Yes           NA           NA
##      Q118232.fctr Q118117.fctr Q117193.fctr Q117186.fctr Q116797.fctr
## 3012           Pr           No           NA           NA           NA
## 5348           NA           NA           NA           NA           NA
##      Q116881.fctr Q116953.fctr Q116601.fctr Q116441.fctr Q116448.fctr
## 3012           NA           NA           NA           NA           NA
## 5348           NA           NA           NA           NA           NA
##      Q116197.fctr Q115602.fctr Q115777.fctr Q115610.fctr Q115611.fctr
## 3012           NA          Yes        Start          Yes          Yes
## 5348           NA           NA           NA           NA           NA
##      Q115899.fctr Q115390.fctr Q115195.fctr Q114961.fctr Q114748.fctr
## 3012           Me           No           No           No           No
## 5348           NA           NA           NA           NA           NA
##      Q114517.fctr Q114386.fctr Q114152.fctr Q113992.fctr Q113583.fctr
## 3012           No          TMI          Yes           No        Tunes
## 5348           NA           NA           NA           NA           NA
##      Q113584.fctr Q113181.fctr Q112478.fctr Q112512.fctr Q112270.fctr
## 3012       People           NA           NA           NA           No
## 5348           NA           NA           NA           NA           NA
##      Q111848.fctr Q111580.fctr Q111220.fctr Q110740.fctr Q109367.fctr
## 3012          Yes    Demanding           No           PC           NA
## 5348           NA           NA           NA           NA           NA
##      Q109244.fctr Q108950.fctr Q108855.fctr Q108617.fctr Q108856.fctr
## 3012           NA           NA           NA           NA           NA
## 5348           NA           NA           NA           NA           NA
##      Q108754.fctr Q108342.fctr Q108343.fctr Q107869.fctr Q107491.fctr
## 3012           NA           NA           NA           NA           NA
## 5348           NA           NA           NA           NA           NA
##      Q106993.fctr Q106997.fctr Q106272.fctr Q106388.fctr Q106389.fctr
## 3012           NA           NA           NA           NA           NA
## 5348           NA           NA           NA           NA           NA
##      Q106042.fctr Q105840.fctr Q105655.fctr Q104996.fctr Q103293.fctr
## 3012           NA          Yes          Yes          Yes           No
## 5348           NA           NA           NA           NA           NA
##      Q102906.fctr Q102674.fctr Q102687.fctr Q102289.fctr Q102089.fctr
## 3012           NA           NA           NA           NA          Own
## 5348           NA           NA           NA           NA           NA
##      Q101162.fctr Q101163.fctr Q101596.fctr Q100689.fctr Q100680.fctr
## 3012     Optimist          Dad           No          Yes           No
## 5348           NA           NA           NA           NA           NA
##      Q100562.fctr Q100010.fctr Q99982.fctr Q99716.fctr Q99581.fctr
## 3012           No          Yes      Check!         Yes          No
## 5348           NA           NA          NA          NA          NA
##      Q99480.fctr Q98869.fctr Q98578.fctr Q98197.fctr Q98059.fctr
## 3012          NA          NA          NA          NA          NA
## 5348          NA          NA          NA          NA          NA
##      Q98078.fctr Q96024.fctr
## 3012          NA         Yes
## 5348          NA          NA
## [1] "min distance(0.9500) pair:"
##      USER_ID Party.fctr Hhold.fctr YOB.Age.fctr Gender.fctr Income.fctr
## 4126    5146          D        SKy           NA           M           N
## 6840    6393       <NA>        SKy           NA           M           N
##      Edn.fctr Q124742.fctr Q124122.fctr Q123621.fctr Q123464.fctr
## 4126      HSD           NA           NA           NA           NA
## 6840        N           NA           NA           NA           NA
##      Q122771.fctr Q122770.fctr Q122769.fctr Q122120.fctr Q121700.fctr
## 4126           NA           NA           NA           NA           NA
## 6840           NA           NA           NA           NA           NA
##      Q121699.fctr Q121011.fctr Q120978.fctr Q120650.fctr Q120472.fctr
## 4126           NA           NA           NA           NA           NA
## 6840           NA           NA           NA           NA           NA
##      Q120379.fctr Q120194.fctr Q120014.fctr Q120012.fctr Q119851.fctr
## 4126           NA           NA           NA           NA           NA
## 6840           NA           NA           NA           NA           NA
##      Q119650.fctr Q119334.fctr Q118892.fctr Q118237.fctr Q118233.fctr
## 4126           NA           NA           NA           NA           NA
## 6840           NA           NA           NA           NA           NA
##      Q118232.fctr Q118117.fctr Q117193.fctr Q117186.fctr Q116797.fctr
## 4126           NA           NA           NA           NA           NA
## 6840           NA           NA           NA           NA           NA
##      Q116881.fctr Q116953.fctr Q116601.fctr Q116441.fctr Q116448.fctr
## 4126           NA           NA           NA           NA           NA
## 6840           NA           NA          Yes           No          Yes
##      Q116197.fctr Q115602.fctr Q115777.fctr Q115610.fctr Q115611.fctr
## 4126           NA          Yes           NA           NA           NA
## 6840         P.M.          Yes          End          Yes          Yes
##      Q115899.fctr Q115390.fctr Q115195.fctr Q114961.fctr Q114748.fctr
## 4126           NA           No           No           No          Yes
## 6840           Cs           No          Yes          Yes          Yes
##      Q114517.fctr Q114386.fctr Q114152.fctr Q113992.fctr Q113583.fctr
## 4126          Yes           NA           NA           NA           NA
## 6840           No   Mysterious           NA           NA           NA
##      Q113584.fctr Q113181.fctr Q112478.fctr Q112512.fctr Q112270.fctr
## 4126           NA           NA           NA           NA           NA
## 6840           NA           NA           NA           NA           NA
##      Q111848.fctr Q111580.fctr Q111220.fctr Q110740.fctr Q109367.fctr
## 4126           NA           NA           NA           NA           NA
## 6840           NA           NA           NA           NA           NA
##      Q109244.fctr Q108950.fctr Q108855.fctr Q108617.fctr Q108856.fctr
## 4126           NA           NA           NA           NA           NA
## 6840           NA           NA           NA           NA           NA
##      Q108754.fctr Q108342.fctr Q108343.fctr Q107869.fctr Q107491.fctr
## 4126           NA           NA           NA           NA           NA
## 6840           NA           NA           NA           NA           NA
##      Q106993.fctr Q106997.fctr Q106272.fctr Q106388.fctr Q106389.fctr
## 4126           NA           NA           NA           NA           NA
## 6840           NA           NA           NA           NA           NA
##      Q106042.fctr Q105840.fctr Q105655.fctr Q104996.fctr Q103293.fctr
## 4126           NA           NA           NA           NA           NA
## 6840           NA           NA           NA           NA           NA
##      Q102906.fctr Q102674.fctr Q102687.fctr Q102289.fctr Q102089.fctr
## 4126           NA           NA           NA           NA           NA
## 6840           NA           NA           NA           NA           NA
##      Q101162.fctr Q101163.fctr Q101596.fctr Q100689.fctr Q100680.fctr
## 4126           NA           NA           NA           NA           NA
## 6840           NA           NA           NA           NA           NA
##      Q100562.fctr Q100010.fctr Q99982.fctr Q99716.fctr Q99581.fctr
## 4126           NA           NA          NA          NA          NA
## 6840           NA           NA          NA          NA          NA
##      Q99480.fctr Q98869.fctr Q98578.fctr Q98197.fctr Q98059.fctr
## 4126          NA          NA          NA          NA          NA
## 6840          NA          NA          NA          NA          NA
##      Q98078.fctr Q96024.fctr
## 4126          NA          NA
## 6840          NA          NA
##    Hhold.fctr .clusterid Hhold.fctr.clusterid   R   D  .entropy .knt
## 1           N          1                  N_1  95 106 0.6916489  201
## 2           N          2                  N_2  77  76 0.6931258  153
## 3           N          3                  N_3  48  48 0.6931472   96
## 4         MKn          1                MKn_1 205 194 0.6927671  399
## 5         MKn          2                MKn_2  75  89 0.6894991  164
## 6         MKn          3                MKn_3  28  61 0.6227371   89
## 7         MKy          1                MKy_1 620 438 0.6782774 1058
## 8         MKy          2                MKy_2 134 136 0.6931197  270
## 9         MKy          3                MKy_3  88 178 0.6347628  266
## 10        PKn          1                PKn_1  27  69 0.5941300   96
## 11        PKn          2                PKn_2  10  46 0.4692203   56
## 12        PKn          3                PKn_3  12  16 0.6829081   28
## 13        PKy          1                PKy_1   8   9 0.6914161   17
## 14        PKy          2                PKy_2  10   8 0.6869616   18
## 15        PKy          3                PKy_3   7   5 0.6791933   12
## 16        PKy          4                PKy_4   1  13 0.2573186   14
## 17        SKn          1                SKn_1 512 456 0.6914729  968
## 18        SKn          2                SKn_2 257 475 0.6481206  732
## 19        SKn          3                SKn_3 167 200 0.6890991  367
## 20        SKn          4                SKn_4 155 209 0.6821024  364
## 21        SKy          1                SKy_1  45  86 0.6433372  131
## 22        SKy          2                SKy_2  21  17 0.6875967   38
## 23        SKy          3                SKy_3  15  16 0.6926268   31
## [1] "glbObsAll$Hhold.fctr$.clusterid Entropy: 0.6726 (98.0608 pct)"

##                     label step_major step_minor label_minor     bgn
## 1            cluster.data          1          0           0   9.146
## 2 partition.data.training          2          0           0 137.717
##       end elapsed
## 1 137.716  128.57
## 2      NA      NA

Step `2.0: partition data training`

## [1] "partition.data.training chunk: setup: elapsed: 0.00 secs"

## Loading required package: sqldf

## Loading required package: gsubfn

## Loading required package: proto

## Loading required package: RSQLite

## Loading required package: DBI

## Loading required package: tcltk

## Loading required package: reshape2

## [1] "partition.data.training chunk: strata_mtrx complete: elapsed: 5.26 secs"
## [1] "partition.data.training chunk: obs_freq_df complete: elapsed: 5.26 secs"

## Loading required package: sampling

## 
## Attaching package: 'sampling'

## The following object is masked from 'package:caret':
## 
##     cluster

## [1] "lclgetMatrixCorrelation: duration: 42.063000 secs"
## [1] "cor of Fit vs. OOB: 1.0000"
## [1] "lclgetMatrixCorrelation: duration: 14.845000 secs"
## [1] "cor of New vs. OOB: 1.0000"
## [1] "lclgetMatrixCorrelation: duration: 51.758000 secs"
## [1] "cor of Fit vs. New: 1.0000"

## [1] "partition.data.training chunk: Fit/OOB partition complete: elapsed: 114.85 secs"

##     Party.Democrat Party.Republican Party.NA
##                 NA               NA     1392
## Fit           2357             2091       NA
## OOB            594              526       NA
##     Party.Democrat Party.Republican Party.NA
##                 NA               NA        1
## Fit      0.5299011        0.4700989       NA
## OOB      0.5303571        0.4696429       NA
##   Hhold.fctr .n.Fit .n.OOB .n.Tst .freqRatio.Fit .freqRatio.OOB
## 6        SKn   1920    511    638     0.43165468    0.456250000
## 2        MKy   1296    298    371     0.29136691    0.266071429
## 1        MKn    516    136    169     0.11600719    0.121428571
## 3          N    367     83    102     0.08250899    0.074107143
## 7        SKy    147     53     65     0.03304856    0.047321429
## 4        PKn    150     30     37     0.03372302    0.026785714
## 5        PKy     52      9     10     0.01169065    0.008035714
##   .freqRatio.Tst
## 6    0.458333333
## 2    0.266522989
## 1    0.121408046
## 3    0.073275862
## 7    0.046695402
## 4    0.026580460
## 5    0.007183908

## [1] "glbObsAll: "

## [1] 6960  221

## [1] "glbObsTrn: "

## [1] 5568  221

## [1] "glbObsFit: "

## [1] 4448  220

## [1] "glbObsOOB: "

## [1] 1120  220

## [1] "glbObsNew: "

## [1] 1392  220

## [1] "partition.data.training chunk: teardown: elapsed: 115.78 secs"

##                     label step_major step_minor label_minor     bgn
## 2 partition.data.training          2          0           0 137.717
## 3         select.features          3          0           0 253.568
##       end elapsed
## 2 253.567  115.85
## 3      NA      NA

Step `3.0: select features`

## [1] "cor(Q98059.fctr, Q98078.fctr)=0.7689"
## [1] "cor(Party.fctr, Q98059.fctr)=0.0172"
## [1] "cor(Party.fctr, Q98078.fctr)=0.0257"

## Warning in myfind_cor_features(feats_df = glb_feats_df, obs_df =
## glbObsTrn, : Identified Q98059.fctr as highly correlated with Q98078.fctr

## [1] "cor(Q99480.fctr, Q99581.fctr)=0.7660"
## [1] "cor(Party.fctr, Q99480.fctr)=-0.0344"
## [1] "cor(Party.fctr, Q99581.fctr)=-0.0104"

## Warning in myfind_cor_features(feats_df = glb_feats_df, obs_df =
## glbObsTrn, : Identified Q99581.fctr as highly correlated with Q99480.fctr

## [1] "cor(Q108855.fctr, Q108856.fctr)=0.7430"
## [1] "cor(Party.fctr, Q108855.fctr)=-0.0371"
## [1] "cor(Party.fctr, Q108856.fctr)=-0.0140"

## Warning in myfind_cor_features(feats_df = glb_feats_df, obs_df =
## glbObsTrn, : Identified Q108856.fctr as highly correlated with Q108855.fctr

## [1] "cor(Q122770.fctr, Q122771.fctr)=0.7379"
## [1] "cor(Party.fctr, Q122770.fctr)=-0.0195"
## [1] "cor(Party.fctr, Q122771.fctr)=-0.0348"

## Warning in myfind_cor_features(feats_df = glb_feats_df, obs_df =
## glbObsTrn, : Identified Q122770.fctr as highly correlated with Q122771.fctr

## [1] "cor(Q106272.fctr, Q106388.fctr)=0.7339"
## [1] "cor(Party.fctr, Q106272.fctr)=-0.0401"
## [1] "cor(Party.fctr, Q106388.fctr)=-0.0342"

## Warning in myfind_cor_features(feats_df = glb_feats_df, obs_df =
## glbObsTrn, : Identified Q106388.fctr as highly correlated with Q106272.fctr

## [1] "cor(Q100680.fctr, Q100689.fctr)=0.7292"
## [1] "cor(Party.fctr, Q100680.fctr)=0.0158"
## [1] "cor(Party.fctr, Q100689.fctr)=0.0257"

## Warning in myfind_cor_features(feats_df = glb_feats_df, obs_df =
## glbObsTrn, : Identified Q100680.fctr as highly correlated with Q100689.fctr

## [1] "cor(Q99480.fctr, Q99716.fctr)=0.7252"
## [1] "cor(Party.fctr, Q99480.fctr)=-0.0344"
## [1] "cor(Party.fctr, Q99716.fctr)=0.0209"

## Warning in myfind_cor_features(feats_df = glb_feats_df, obs_df =
## glbObsTrn, : Identified Q99716.fctr as highly correlated with Q99480.fctr

## [1] "cor(Q120472.fctr, Q120650.fctr)=0.7126"
## [1] "cor(Party.fctr, Q120472.fctr)=-0.0462"
## [1] "cor(Party.fctr, Q120650.fctr)=-0.0271"

## Warning in myfind_cor_features(feats_df = glb_feats_df, obs_df =
## glbObsTrn, : Identified Q120650.fctr as highly correlated with Q120472.fctr

## [1] "cor(Q98869.fctr, Q99480.fctr)=0.7084"
## [1] "cor(Party.fctr, Q98869.fctr)=-0.0277"
## [1] "cor(Party.fctr, Q99480.fctr)=-0.0344"

## Warning in myfind_cor_features(feats_df = glb_feats_df, obs_df =
## glbObsTrn, : Identified Q98869.fctr as highly correlated with Q99480.fctr

## [1] "cor(Q123464.fctr, Q123621.fctr)=0.7078"
## [1] "cor(Party.fctr, Q123464.fctr)=-0.0136"
## [1] "cor(Party.fctr, Q123621.fctr)=-0.0255"

## Warning in myfind_cor_features(feats_df = glb_feats_df, obs_df =
## glbObsTrn, : Identified Q123464.fctr as highly correlated with Q123621.fctr

## [1] "cor(Q108754.fctr, Q108855.fctr)=0.7005"
## [1] "cor(Party.fctr, Q108754.fctr)=-0.0081"
## [1] "cor(Party.fctr, Q108855.fctr)=-0.0371"

## Warning in myfind_cor_features(feats_df = glb_feats_df, obs_df =
## glbObsTrn, : Identified Q108754.fctr as highly correlated with Q108855.fctr

##                         cor.y exclude.as.feat    cor.y.abs   cor.high.X
## Q109244.fctr     0.1203812469               0 0.1203812469         <NA>
## .clusterid       0.0984277178               1 0.0984277178         <NA>
## .clusterid.fctr  0.0984277178               0 0.0984277178         <NA>
## Hhold.fctr       0.0511386673               0 0.0511386673         <NA>
## Edn.fctr         0.0359295351               0 0.0359295351         <NA>
## Q101163.fctr     0.0295046473               0 0.0295046473         <NA>
## Q100689.fctr     0.0256915080               0 0.0256915080         <NA>
## Q98078.fctr      0.0256516490               0 0.0256516490         <NA>
## Q99716.fctr      0.0209286674               0 0.0209286674  Q99480.fctr
## Q120379.fctr     0.0206291292               0 0.0206291292         <NA>
## Q121699.fctr     0.0196933075               0 0.0196933075         <NA>
## Q105840.fctr     0.0195569165               0 0.0195569165         <NA>
## Q113583.fctr     0.0191894717               0 0.0191894717         <NA>
## Q115195.fctr     0.0174522586               0 0.0174522586         <NA>
## Q102089.fctr     0.0174087944               0 0.0174087944         <NA>
## Q98059.fctr      0.0171637755               0 0.0171637755  Q98078.fctr
## Q114386.fctr     0.0168013326               0 0.0168013326         <NA>
## Q100680.fctr     0.0157762454               0 0.0157762454 Q100689.fctr
## Q108342.fctr     0.0151842510               0 0.0151842510         <NA>
## Q111848.fctr     0.0141099384               0 0.0141099384         <NA>
## YOB.Age.fctr     0.0129198495               0 0.0129198495         <NA>
## Q118892.fctr     0.0125250379               0 0.0125250379         <NA>
## Q102687.fctr     0.0120079165               0 0.0120079165         <NA>
## Q115390.fctr     0.0119300319               0 0.0119300319         <NA>
## Q119851.fctr     0.0093381833               0 0.0093381833         <NA>
## Q114517.fctr     0.0084741753               0 0.0084741753         <NA>
## Q120012.fctr     0.0084652930               0 0.0084652930         <NA>
## Q109367.fctr     0.0080456026               0 0.0080456026         <NA>
## Q114961.fctr     0.0079206587               0 0.0079206587         <NA>
## Q121700.fctr     0.0067756198               0 0.0067756198         <NA>
## Q124122.fctr     0.0061257448               0 0.0061257448         <NA>
## Q111220.fctr     0.0055758571               0 0.0055758571         <NA>
## Q113992.fctr     0.0041479796               0 0.0041479796         <NA>
## Q121011.fctr     0.0037329030               0 0.0037329030         <NA>
## Q106042.fctr     0.0032327194               0 0.0032327194         <NA>
## Q116448.fctr     0.0031731051               0 0.0031731051         <NA>
## Q116601.fctr     0.0022379241               0 0.0022379241         <NA>
## Q104996.fctr     0.0012202806               0 0.0012202806         <NA>
## Q102906.fctr     0.0011540297               0 0.0011540297         <NA>
## Q113584.fctr     0.0011387024               0 0.0011387024         <NA>
## Q108950.fctr     0.0010567028               0 0.0010567028         <NA>
## Q102674.fctr     0.0009759844               0 0.0009759844         <NA>
## Q103293.fctr     0.0005915534               0 0.0005915534         <NA>
## Q112478.fctr     0.0001517248               0 0.0001517248         <NA>
## Q114748.fctr    -0.0008477228               0 0.0008477228         <NA>
## Q107491.fctr    -0.0014031814               0 0.0014031814         <NA>
## Q100562.fctr    -0.0017132769               0 0.0017132769         <NA>
## Q108617.fctr    -0.0024119725               0 0.0024119725         <NA>
## Q100010.fctr    -0.0024291540               0 0.0024291540         <NA>
## Q115602.fctr    -0.0027844465               0 0.0027844465         <NA>
## Q116953.fctr    -0.0029786716               0 0.0029786716         <NA>
## Q115610.fctr    -0.0035255582               0 0.0035255582         <NA>
## Q106997.fctr    -0.0041749086               0 0.0041749086         <NA>
## Q120978.fctr    -0.0044187616               0 0.0044187616         <NA>
## Q112512.fctr    -0.0056768212               0 0.0056768212         <NA>
## Q108343.fctr    -0.0060665340               0 0.0060665340         <NA>
## Q96024.fctr     -0.0069116541               0 0.0069116541         <NA>
## Q106389.fctr    -0.0077498918               0 0.0077498918         <NA>
## .rnorm          -0.0078039520               0 0.0078039520         <NA>
## Q108754.fctr    -0.0080847764               0 0.0080847764 Q108855.fctr
## Q98578.fctr     -0.0081164509               0 0.0081164509         <NA>
## Q101162.fctr    -0.0099412952               0 0.0099412952         <NA>
## Q115777.fctr    -0.0101315203               0 0.0101315203         <NA>
## Q99581.fctr     -0.0103662478               0 0.0103662478  Q99480.fctr
## Q124742.fctr    -0.0111642906               0 0.0111642906         <NA>
## Q116797.fctr    -0.0112749656               0 0.0112749656         <NA>
## Q112270.fctr    -0.0116157798               0 0.0116157798         <NA>
## YOB             -0.0116828198               1 0.0116828198         <NA>
## Q118237.fctr    -0.0117079669               0 0.0117079669         <NA>
## Q119650.fctr    -0.0125645475               0 0.0125645475         <NA>
## Q111580.fctr    -0.0132382335               0 0.0132382335         <NA>
## Q123464.fctr    -0.0136140083               0 0.0136140083 Q123621.fctr
## Q117193.fctr    -0.0138241599               0 0.0138241599         <NA>
## Q99982.fctr     -0.0139727928               0 0.0139727928         <NA>
## Q108856.fctr    -0.0140363785               0 0.0140363785 Q108855.fctr
## Q118233.fctr    -0.0147269325               0 0.0147269325         <NA>
## Q102289.fctr    -0.0155850393               0 0.0155850393         <NA>
## Q116197.fctr    -0.0158561766               0 0.0158561766         <NA>
## Income.fctr     -0.0159635458               0 0.0159635458         <NA>
## Q118232.fctr    -0.0171321152               0 0.0171321152         <NA>
## Q120194.fctr    -0.0172986920               0 0.0172986920         <NA>
## Q114152.fctr    -0.0175013163               0 0.0175013163         <NA>
## Q122770.fctr    -0.0194639697               0 0.0194639697 Q122771.fctr
## Q117186.fctr    -0.0198853672               0 0.0198853672         <NA>
## Q105655.fctr    -0.0198994078               0 0.0198994078         <NA>
## Q106993.fctr    -0.0207428635               0 0.0207428635         <NA>
## Q119334.fctr    -0.0226894034               0 0.0226894034         <NA>
## Q122120.fctr    -0.0229287700               0 0.0229287700         <NA>
## Q116441.fctr    -0.0237358205               0 0.0237358205         <NA>
## Q118117.fctr    -0.0253544150               0 0.0253544150         <NA>
## Q123621.fctr    -0.0255329743               0 0.0255329743         <NA>
## Q122769.fctr    -0.0259739146               0 0.0259739146         <NA>
## Q120650.fctr    -0.0270889067               0 0.0270889067 Q120472.fctr
## Q98869.fctr     -0.0276734114               0 0.0276734114  Q99480.fctr
## .pos            -0.0302037138               1 0.0302037138         <NA>
## USER_ID         -0.0302304868               1 0.0302304868         <NA>
## Q107869.fctr    -0.0304661021               0 0.0304661021         <NA>
## Q120014.fctr    -0.0318620439               0 0.0318620439         <NA>
## Q115899.fctr    -0.0324177950               0 0.0324177950         <NA>
## Q106388.fctr    -0.0341579350               0 0.0341579350 Q106272.fctr
## Q99480.fctr     -0.0344412239               0 0.0344412239         <NA>
## Q122771.fctr    -0.0348421015               0 0.0348421015         <NA>
## Q108855.fctr    -0.0370970211               0 0.0370970211         <NA>
## Q110740.fctr    -0.0380691243               0 0.0380691243         <NA>
## Q106272.fctr    -0.0400926462               0 0.0400926462         <NA>
## Q101596.fctr    -0.0409784077               0 0.0409784077         <NA>
## Q116881.fctr    -0.0416860293               0 0.0416860293         <NA>
## Q120472.fctr    -0.0462030674               0 0.0462030674         <NA>
## Q98197.fctr     -0.0549342527               0 0.0549342527         <NA>
## Q113181.fctr    -0.0808753072               0 0.0808753072         <NA>
## Q115611.fctr    -0.0904468203               0 0.0904468203         <NA>
## Gender.fctr     -0.1027400851               0 0.1027400851         <NA>
##                 freqRatio percentUnique zeroVar   nzv is.cor.y.abs.low
## Q109244.fctr     1.125916    0.05387931   FALSE FALSE            FALSE
## .clusterid       2.005590    0.07183908   FALSE FALSE            FALSE
## .clusterid.fctr  2.005590    0.07183908   FALSE FALSE            FALSE
## Hhold.fctr       1.525094    0.12571839   FALSE FALSE            FALSE
## Edn.fctr         1.392610    0.14367816   FALSE FALSE            FALSE
## Q101163.fctr     1.327394    0.05387931   FALSE FALSE            FALSE
## Q100689.fctr     1.029800    0.05387931   FALSE FALSE            FALSE
## Q98078.fctr      1.266595    0.05387931   FALSE FALSE            FALSE
## Q99716.fctr      1.328693    0.05387931   FALSE FALSE            FALSE
## Q120379.fctr     1.046326    0.05387931   FALSE FALSE            FALSE
## Q121699.fctr     1.507127    0.05387931   FALSE FALSE            FALSE
## Q105840.fctr     1.275362    0.05387931   FALSE FALSE            FALSE
## Q113583.fctr     1.102515    0.05387931   FALSE FALSE            FALSE
## Q115195.fctr     1.065496    0.05387931   FALSE FALSE            FALSE
## Q102089.fctr     1.055963    0.05387931   FALSE FALSE            FALSE
## Q98059.fctr      1.493810    0.05387931   FALSE FALSE            FALSE
## Q114386.fctr     1.092072    0.05387931   FALSE FALSE            FALSE
## Q100680.fctr     1.102386    0.05387931   FALSE FALSE            FALSE
## Q108342.fctr     1.048292    0.05387931   FALSE FALSE            FALSE
## Q111848.fctr     1.113602    0.05387931   FALSE FALSE            FALSE
## YOB.Age.fctr     1.005794    0.16163793   FALSE FALSE            FALSE
## Q118892.fctr     1.347380    0.05387931   FALSE FALSE            FALSE
## Q102687.fctr     1.256545    0.05387931   FALSE FALSE            FALSE
## Q115390.fctr     1.150505    0.05387931   FALSE FALSE            FALSE
## Q119851.fctr     1.244519    0.05387931   FALSE FALSE            FALSE
## Q114517.fctr     1.183374    0.05387931   FALSE FALSE            FALSE
## Q120012.fctr     1.047185    0.05387931   FALSE FALSE            FALSE
## Q109367.fctr     1.008571    0.05387931   FALSE FALSE            FALSE
## Q114961.fctr     1.250436    0.05387931   FALSE FALSE            FALSE
## Q121700.fctr     1.708221    0.05387931   FALSE FALSE             TRUE
## Q124122.fctr     1.412807    0.05387931   FALSE FALSE             TRUE
## Q111220.fctr     1.262849    0.05387931   FALSE FALSE             TRUE
## Q113992.fctr     1.267442    0.05387931   FALSE FALSE             TRUE
## Q121011.fctr     1.153676    0.05387931   FALSE FALSE             TRUE
## Q106042.fctr     1.247738    0.05387931   FALSE FALSE             TRUE
## Q116448.fctr     1.161031    0.05387931   FALSE FALSE             TRUE
## Q116601.fctr     1.394914    0.05387931   FALSE FALSE             TRUE
## Q104996.fctr     1.173840    0.05387931   FALSE FALSE             TRUE
## Q102906.fctr     1.053396    0.05387931   FALSE FALSE             TRUE
## Q113584.fctr     1.212486    0.05387931   FALSE FALSE             TRUE
## Q108950.fctr     1.103872    0.05387931   FALSE FALSE             TRUE
## Q102674.fctr     1.073412    0.05387931   FALSE FALSE             TRUE
## Q103293.fctr     1.122287    0.05387931   FALSE FALSE             TRUE
## Q112478.fctr     1.113648    0.05387931   FALSE FALSE             TRUE
## Q114748.fctr     1.051125    0.05387931   FALSE FALSE             TRUE
## Q107491.fctr     1.419021    0.05387931   FALSE FALSE             TRUE
## Q100562.fctr     1.217215    0.05387931   FALSE FALSE             TRUE
## Q108617.fctr     1.390618    0.05387931   FALSE FALSE             TRUE
## Q100010.fctr     1.268156    0.05387931   FALSE FALSE             TRUE
## Q115602.fctr     1.322302    0.05387931   FALSE FALSE             TRUE
## Q116953.fctr     1.039180    0.05387931   FALSE FALSE             TRUE
## Q115610.fctr     1.359695    0.05387931   FALSE FALSE             TRUE
## Q106997.fctr     1.177632    0.05387931   FALSE FALSE             TRUE
## Q120978.fctr     1.131963    0.05387931   FALSE FALSE             TRUE
## Q112512.fctr     1.299253    0.05387931   FALSE FALSE             TRUE
## Q108343.fctr     1.064910    0.05387931   FALSE FALSE             TRUE
## Q96024.fctr      1.144428    0.05387931   FALSE FALSE             TRUE
## Q106389.fctr     1.341307    0.05387931   FALSE FALSE             TRUE
## .rnorm           1.000000  100.00000000   FALSE FALSE            FALSE
## Q108754.fctr     1.008090    0.05387931   FALSE FALSE            FALSE
## Q98578.fctr      1.093556    0.05387931   FALSE FALSE            FALSE
## Q101162.fctr     1.103229    0.05387931   FALSE FALSE            FALSE
## Q115777.fctr     1.140288    0.05387931   FALSE FALSE            FALSE
## Q99581.fctr      1.375000    0.05387931   FALSE FALSE            FALSE
## Q124742.fctr     2.565379    0.05387931   FALSE FALSE            FALSE
## Q116797.fctr     1.009589    0.05387931   FALSE FALSE            FALSE
## Q112270.fctr     1.254284    0.05387931   FALSE FALSE            FALSE
## YOB              1.027559    1.41882184   FALSE FALSE            FALSE
## Q118237.fctr     1.088017    0.05387931   FALSE FALSE            FALSE
## Q119650.fctr     1.456978    0.05387931   FALSE FALSE            FALSE
## Q111580.fctr     1.024977    0.05387931   FALSE FALSE            FALSE
## Q123464.fctr     1.326681    0.05387931   FALSE FALSE            FALSE
## Q117193.fctr     1.140665    0.05387931   FALSE FALSE            FALSE
## Q99982.fctr      1.339380    0.05387931   FALSE FALSE            FALSE
## Q108856.fctr     1.080645    0.05387931   FALSE FALSE            FALSE
## Q118233.fctr     1.199142    0.05387931   FALSE FALSE            FALSE
## Q102289.fctr     1.033482    0.05387931   FALSE FALSE            FALSE
## Q116197.fctr     1.073778    0.05387931   FALSE FALSE            FALSE
## Income.fctr      1.256724    0.12571839   FALSE FALSE            FALSE
## Q118232.fctr     1.365812    0.05387931   FALSE FALSE            FALSE
## Q120194.fctr     1.016716    0.05387931   FALSE FALSE            FALSE
## Q114152.fctr     1.027617    0.05387931   FALSE FALSE            FALSE
## Q122770.fctr     1.008802    0.05387931   FALSE FALSE            FALSE
## Q117186.fctr     1.053878    0.05387931   FALSE FALSE            FALSE
## Q105655.fctr     1.079316    0.05387931   FALSE FALSE            FALSE
## Q106993.fctr     1.327392    0.05387931   FALSE FALSE            FALSE
## Q119334.fctr     1.081498    0.05387931   FALSE FALSE            FALSE
## Q122120.fctr     1.297443    0.05387931   FALSE FALSE            FALSE
## Q116441.fctr     1.019645    0.05387931   FALSE FALSE            FALSE
## Q118117.fctr     1.174006    0.05387931   FALSE FALSE            FALSE
## Q123621.fctr     1.466381    0.05387931   FALSE FALSE            FALSE
## Q122769.fctr     1.060606    0.05387931   FALSE FALSE            FALSE
## Q120650.fctr     1.896247    0.05387931   FALSE FALSE            FALSE
## Q98869.fctr      1.080860    0.05387931   FALSE FALSE            FALSE
## .pos             1.000000  100.00000000   FALSE FALSE            FALSE
## USER_ID          1.000000  100.00000000   FALSE FALSE            FALSE
## Q107869.fctr     1.211050    0.05387931   FALSE FALSE            FALSE
## Q120014.fctr     1.044944    0.05387931   FALSE FALSE            FALSE
## Q115899.fctr     1.197849    0.05387931   FALSE FALSE            FALSE
## Q106388.fctr     1.065033    0.05387931   FALSE FALSE            FALSE
## Q99480.fctr      1.225404    0.05387931   FALSE FALSE            FALSE
## Q122771.fctr     1.414753    0.05387931   FALSE FALSE            FALSE
## Q108855.fctr     1.273980    0.05387931   FALSE FALSE            FALSE
## Q110740.fctr     1.050779    0.05387931   FALSE FALSE            FALSE
## Q106272.fctr     1.116536    0.05387931   FALSE FALSE            FALSE
## Q101596.fctr     1.041667    0.05387931   FALSE FALSE            FALSE
## Q116881.fctr     1.010066    0.05387931   FALSE FALSE            FALSE
## Q120472.fctr     1.292633    0.05387931   FALSE FALSE            FALSE
## Q98197.fctr      1.129371    0.05387931   FALSE FALSE            FALSE
## Q113181.fctr     1.006354    0.05387931   FALSE FALSE            FALSE
## Q115611.fctr     1.194859    0.05387931   FALSE FALSE            FALSE
## Gender.fctr      1.561033    0.05387931   FALSE FALSE            FALSE

## Warning in myplot_scatter(plt_feats_df, "percentUnique", "freqRatio",
## colorcol_name = "nzv", : converting nzv to class:factor

## Warning: Removed 3 rows containing missing values (geom_point).

## Warning: Removed 3 rows containing missing values (geom_point).

## Warning: Removed 3 rows containing missing values (geom_point).

## [1] cor.y            exclude.as.feat  cor.y.abs        cor.high.X      
## [5] freqRatio        percentUnique    zeroVar          nzv             
## [9] is.cor.y.abs.low
## <0 rows> (or 0-length row.names)

## Scale for 'y' is already present. Adding another scale for 'y', which
## will replace the existing scale.

## [1] "numeric data missing in glbObsAll: "
##        YOB Party.fctr 
##        415       1392 
## [1] "numeric data w/ 0s in glbObsAll: "
## named integer(0)
## [1] "numeric data w/ Infs in glbObsAll: "
## named integer(0)
## [1] "numeric data w/ NaNs in glbObsAll: "
## named integer(0)
## [1] "string data missing in glbObsAll: "
##          Gender          Income HouseholdStatus  EducationLevel 
##             143            1273             552            1067 
##           Party         Q124742         Q124122         Q123464 
##              NA            4340            3114            2912 
##         Q123621         Q122769         Q122770         Q122771 
##            3018            2778            2597            2579 
##         Q122120         Q121699         Q121700         Q120978 
##            2552            2279            2328            2303 
##         Q121011         Q120379         Q120650         Q120472 
##            2256            2361            2283            2433 
##         Q120194         Q120012         Q120014         Q119334 
##            2603            2344            2571            2477 
##         Q119851         Q119650         Q118892         Q118117 
##            2243            2374            2206            2342 
##         Q118232         Q118233         Q118237         Q117186 
##            3018            2659            2592            2845 
##         Q117193         Q116797         Q116881         Q116953 
##            2799            2771            2889            2848 
##         Q116601         Q116441         Q116448         Q116197 
##            2606            2684            2730            2657 
##         Q115602         Q115777         Q115610         Q115611 
##            2619            2785            2637            2443 
##         Q115899         Q115390         Q114961         Q114748 
##            2789            2860            2687            2462 
##         Q115195         Q114517         Q114386         Q113992 
##            2647            2567            2686            2502 
##         Q114152         Q113583         Q113584         Q113181 
##            2829            2632            2654            2576 
##         Q112478         Q112512         Q112270         Q111848 
##            2790            2676            2820            2449 
##         Q111580         Q111220         Q110740         Q109367 
##            2686            2563            2479            2624 
##         Q108950         Q109244         Q108855         Q108617 
##            2641            2731            3008            2696 
##         Q108856         Q108754         Q108342         Q108343 
##            3007            2770            2760            2736 
##         Q107869         Q107491         Q106993         Q106997 
##            2762            2667            2676            2702 
##         Q106272         Q106388         Q106389         Q106042 
##            2722            2818            2871            2762 
##         Q105840         Q105655         Q104996         Q103293 
##            2876            2612            2620            2674 
##         Q102906         Q102674         Q102687         Q102289 
##            2840            2864            2712            2790 
##         Q102089         Q101162         Q101163         Q101596 
##            2736            2816            2995            2824 
##         Q100689         Q100680         Q100562          Q99982 
##            2568            2787            2793            2871 
##         Q100010          Q99716          Q99581          Q99480 
##            2688            2790            2690            2700 
##          Q98869          Q98578          Q98059          Q98078 
##            2906            2867            2629            2945 
##          Q98197          Q96024            .lcn 
##            2836            2858            1392

## [1] "glb_feats_df:"

## [1] 112  12

##                    id exclude.as.feat rsp_var
## Party.fctr Party.fctr            TRUE    TRUE

##                    id       cor.y exclude.as.feat  cor.y.abs cor.high.X
## USER_ID       USER_ID -0.03023049            TRUE 0.03023049       <NA>
## Party.fctr Party.fctr          NA            TRUE         NA       <NA>
##            freqRatio percentUnique zeroVar   nzv is.cor.y.abs.low
## USER_ID            1           100   FALSE FALSE            FALSE
## Party.fctr        NA            NA      NA    NA               NA
##            interaction.feat shapiro.test.p.value rsp_var_raw id_var
## USER_ID                <NA>                   NA       FALSE   TRUE
## Party.fctr             <NA>                   NA          NA     NA
##            rsp_var
## USER_ID         NA
## Party.fctr    TRUE

## [1] "glb_feats_df vs. glbObsAll: "

## character(0)

## [1] "glbObsAll vs. glb_feats_df: "

## character(0)

##             label step_major step_minor label_minor     bgn     end
## 3 select.features          3          0           0 253.568 259.548
## 4      fit.models          4          0           0 259.549      NA
##   elapsed
## 3   5.981
## 4      NA

Step `4.0: fit models`

fit.models_0_chunk_df <- myadd_chunk(NULL, "fit.models_0_bgn", label.minor = "setup")

##              label step_major step_minor label_minor     bgn end elapsed
## 1 fit.models_0_bgn          1          0       setup 260.129  NA      NA

# load(paste0(glbOut$pfx, "dsk.RData"))

get_model_sel_frmla <- function() {
    model_evl_terms <- c(NULL)
    # min.aic.fit might not be avl
    lclMdlEvlCriteria <- 
        glbMdlMetricsEval[glbMdlMetricsEval %in% names(glb_models_df)]
    for (metric in lclMdlEvlCriteria)
        model_evl_terms <- c(model_evl_terms, 
                             ifelse(length(grep("max", metric)) > 0, "-", "+"), metric)
    if (glb_is_classification && glb_is_binomial)
        model_evl_terms <- c(model_evl_terms, "-", "opt.prob.threshold.OOB")
    model_sel_frmla <- as.formula(paste(c("~ ", model_evl_terms), collapse = " "))
    return(model_sel_frmla)
}

get_dsp_models_df <- function() {
    dsp_models_cols <- c("id", 
                    glbMdlMetricsEval[glbMdlMetricsEval %in% names(glb_models_df)],
                    grep("opt.", names(glb_models_df), fixed = TRUE, value = TRUE)) 
    dsp_models_df <- 
        #orderBy(get_model_sel_frmla(), glb_models_df)[, c("id", glbMdlMetricsEval)]
        orderBy(get_model_sel_frmla(), glb_models_df)[, dsp_models_cols]    
    nCvMdl <- sapply(glb_models_lst, function(mdl) nrow(mdl$results))
    nParams <- sapply(glb_models_lst, function(mdl) ifelse(mdl$method == "custom", 0, 
        nrow(subset(modelLookup(mdl$method), parameter != "parameter"))))
    
#     nCvMdl <- nCvMdl[names(nCvMdl) != "avNNet"]
#     nParams <- nParams[names(nParams) != "avNNet"]    
    
    if (length(cvMdlProblems <- nCvMdl[nCvMdl <= nParams]) > 0) {
        print("Cross Validation issues:")
        warning("Cross Validation issues:")        
        print(cvMdlProblems)
    }
    
    pltMdls <- setdiff(names(nCvMdl), names(cvMdlProblems))
    pltMdls <- setdiff(pltMdls, names(nParams[nParams == 0]))
    
    # length(pltMdls) == 21
    png(paste0(glbOut$pfx, "bestTune.png"), width = 480 * 2, height = 480 * 4)
    grid.newpage()
    pushViewport(viewport(layout = grid.layout(ceiling(length(pltMdls) / 2.0), 2)))
    pltIx <- 1
    for (mdlId in pltMdls) {
        print(ggplot(glb_models_lst[[mdlId]], highBestTune = TRUE) + labs(title = mdlId),   
              vp = viewport(layout.pos.row = ceiling(pltIx / 2.0), 
                            layout.pos.col = ((pltIx - 1) %% 2) + 1))  
        pltIx <- pltIx + 1
    }
    dev.off()

    if (all(row.names(dsp_models_df) != dsp_models_df$id))
        row.names(dsp_models_df) <- dsp_models_df$id
    return(dsp_models_df)
}
#get_dsp_models_df()

if (glb_is_classification && glb_is_binomial && 
        (length(unique(glbObsFit[, glb_rsp_var])) < 2))
    stop("glbObsFit$", glb_rsp_var, ": contains less than 2 unique values: ",
         paste0(unique(glbObsFit[, glb_rsp_var]), collapse=", "))

max_cor_y_x_vars <- orderBy(~ -cor.y.abs, 
        subset(glb_feats_df, (exclude.as.feat == 0) & !nzv & !is.cor.y.abs.low & 
                                is.na(cor.high.X)))[1:2, "id"]
max_cor_y_x_vars <- max_cor_y_x_vars[!is.na(max_cor_y_x_vars)]
if (length(max_cor_y_x_vars) < 2)
    max_cor_y_x_vars <- union(max_cor_y_x_vars, ".pos")

if (!is.null(glb_Baseline_mdl_var)) {
    if ((max_cor_y_x_vars[1] != glb_Baseline_mdl_var) & 
        (glb_feats_df[glb_feats_df$id == max_cor_y_x_vars[1], "cor.y.abs"] > 
         glb_feats_df[glb_feats_df$id == glb_Baseline_mdl_var, "cor.y.abs"]))
        stop(max_cor_y_x_vars[1], " has a higher correlation with ", glb_rsp_var, 
             " than the Baseline var: ", glb_Baseline_mdl_var)
}

glb_model_type <- ifelse(glb_is_regression, "regression", "classification")
    
# Model specs
# c("id.prefix", "method", "type",
#   # trainControl params
#   "preProc.method", "cv.n.folds", "cv.n.repeats", "summary.fn",
#   # train params
#   "metric", "metric.maximize", "tune.df")

# Baseline
if (!is.null(glb_Baseline_mdl_var)) {
    fit.models_0_chunk_df <- myadd_chunk(fit.models_0_chunk_df, 
                            paste0("fit.models_0_", "Baseline"), major.inc = FALSE,
                                    label.minor = "mybaseln_classfr")
    ret_lst <- myfit_mdl(mdl_id="Baseline", 
                         model_method="mybaseln_classfr",
                        indepVar=glb_Baseline_mdl_var,
                        rsp_var=glb_rsp_var,
                        fit_df=glbObsFit, OOB_df=glbObsOOB)
}    

# Most Frequent Outcome "MFO" model: mean(y) for regression
#   Not using caret's nullModel since model stats not avl
#   Cannot use rpart for multinomial classification since it predicts non-MFO
if (glb_is_classification) {
    fit.models_0_chunk_df <- myadd_chunk(fit.models_0_chunk_df, 
                                paste0("fit.models_0_", "MFO"), major.inc = FALSE,
                                        label.minor = "myMFO_classfr")

    ret_lst <- myfit_mdl(mdl_specs_lst = myinit_mdl_specs_lst(mdl_specs_lst = list(
        id.prefix = "MFO", type = glb_model_type, trainControl.method = "none",
        train.method = ifelse(glb_is_regression, "lm", "myMFO_classfr"))),
                            indepVar = ".rnorm", rsp_var = glb_rsp_var,
                            fit_df = glbObsFit, OOB_df = glbObsOOB)

        # "random" model - only for classification; 
        #   none needed for regression since it is same as MFO
    fit.models_0_chunk_df <- myadd_chunk(fit.models_0_chunk_df, 
                                paste0("fit.models_0_", "Random"), major.inc = FALSE,
                                        label.minor = "myrandom_classfr")

#stop(here"); glb2Sav(); all.equal(glb_models_df, sav_models_df)    
    ret_lst <- myfit_mdl(mdl_specs_lst = myinit_mdl_specs_lst(mdl_specs_lst = list(
        id.prefix = "Random", type = glb_model_type, trainControl.method = "none",
        train.method = "myrandom_classfr")),
                        indepVar = ".rnorm", rsp_var = glb_rsp_var,
                        fit_df = glbObsFit, OOB_df = glbObsOOB)
}

##              label step_major step_minor   label_minor     bgn     end
## 1 fit.models_0_bgn          1          0         setup 260.129 260.161
## 2 fit.models_0_MFO          1          1 myMFO_classfr 260.162      NA
##   elapsed
## 1   0.032
## 2      NA
## [1] "myfit_mdl: enter: 0.000000 secs"
## [1] "fitting model: MFO###myMFO_classfr"
## [1] "    indepVar: .rnorm"
## [1] "myfit_mdl: setup complete: 0.518000 secs"
## Fitting parameter = none on full training set
## [1] "in MFO.Classifier$fit"
## [1] "unique.vals:"
## [1] R D
## Levels: R D
## [1] "unique.prob:"
## y
##         D         R 
## 0.5299011 0.4700989 
## [1] "MFO.val:"
## [1] "D"
## [1] "myfit_mdl: train complete: 1.027000 secs"
##             Length Class      Mode     
## unique.vals 2      factor     numeric  
## unique.prob 2      -none-     numeric  
## MFO.val     1      -none-     character
## x.names     1      -none-     character
## xNames      1      -none-     character
## problemType 1      -none-     character
## tuneValue   1      data.frame list     
## obsLevels   2      -none-     character
## [1] "myfit_mdl: train diagnostics complete: 1.029000 secs"

## Loading required namespace: pROC

## [1] "entr MFO.Classifier$predict"
## [1] "exit MFO.Classifier$predict"

## Loading required package: ROCR

## Loading required package: gplots

## 
## Attaching package: 'gplots'

## The following object is masked from 'package:stats':
## 
##     lowess

## [1] "in MFO.Classifier$prob"
##           R         D
## 1 0.5299011 0.4700989
## 2 0.5299011 0.4700989
## 3 0.5299011 0.4700989
## 4 0.5299011 0.4700989
## 5 0.5299011 0.4700989
## 6 0.5299011 0.4700989

##          Prediction
## Reference    R    D
##         R 2091    0
##         D 2357    0
##       Accuracy          Kappa  AccuracyLower  AccuracyUpper   AccuracyNull 
##      0.4700989      0.0000000      0.4553427      0.4848945      0.5299011 
## AccuracyPValue  McnemarPValue 
##      1.0000000      0.0000000 
## [1] "entr MFO.Classifier$predict"
## [1] "exit MFO.Classifier$predict"
## [1] "in MFO.Classifier$prob"
##           R         D
## 1 0.5299011 0.4700989
## 2 0.5299011 0.4700989
## 3 0.5299011 0.4700989
## 4 0.5299011 0.4700989
## 5 0.5299011 0.4700989
## 6 0.5299011 0.4700989

##          Prediction
## Reference   R   D
##         R 526   0
##         D 594   0
##       Accuracy          Kappa  AccuracyLower  AccuracyUpper   AccuracyNull 
##   4.696429e-01   0.000000e+00   4.400805e-01   4.993651e-01   5.303571e-01 
## AccuracyPValue  McnemarPValue 
##   9.999790e-01  9.194240e-131 
## [1] "myfit_mdl: predict complete: 5.804000 secs"
##                    id  feats max.nTuningRuns min.elapsedtime.everything
## 1 MFO###myMFO_classfr .rnorm               0                      0.498
##   min.elapsedtime.final max.AUCpROC.fit max.Sens.fit max.Spec.fit
## 1                 0.004             0.5            0            1
##   max.AUCROCR.fit opt.prob.threshold.fit max.f.score.fit max.Accuracy.fit
## 1             0.5                    0.5       0.6395473        0.4700989
##   max.AccuracyLower.fit max.AccuracyUpper.fit max.Kappa.fit
## 1             0.4553427             0.4848945             0
##   max.AUCpROC.OOB max.Sens.OOB max.Spec.OOB max.AUCROCR.OOB
## 1             0.5            0            1             0.5
##   opt.prob.threshold.OOB max.f.score.OOB max.Accuracy.OOB
## 1                    0.5       0.6391252        0.4696429
##   max.AccuracyLower.OOB max.AccuracyUpper.OOB max.Kappa.OOB
## 1             0.4400805             0.4993651             0
## [1] "myfit_mdl: exit: 5.813000 secs"
##                 label step_major step_minor      label_minor     bgn
## 2    fit.models_0_MFO          1          1    myMFO_classfr 260.162
## 3 fit.models_0_Random          1          2 myrandom_classfr 265.981
##       end elapsed
## 2 265.981   5.819
## 3      NA      NA
## [1] "myfit_mdl: enter: 0.000000 secs"
## [1] "fitting model: Random###myrandom_classfr"
## [1] "    indepVar: .rnorm"
## [1] "myfit_mdl: setup complete: 0.440000 secs"
## Fitting parameter = none on full training set
## [1] "myfit_mdl: train complete: 0.738000 secs"
##             Length Class      Mode     
## unique.vals 2      factor     numeric  
## unique.prob 2      table      numeric  
## xNames      1      -none-     character
## problemType 1      -none-     character
## tuneValue   1      data.frame list     
## obsLevels   2      -none-     character
## [1] "myfit_mdl: train diagnostics complete: 0.739000 secs"
## [1] "in Random.Classifier$prob"

##          Prediction
## Reference    R    D
##         R 2091    0
##         D 2357    0
##       Accuracy          Kappa  AccuracyLower  AccuracyUpper   AccuracyNull 
##      0.4700989      0.0000000      0.4553427      0.4848945      0.5299011 
## AccuracyPValue  McnemarPValue 
##      1.0000000      0.0000000 
## [1] "in Random.Classifier$prob"

##          Prediction
## Reference   R   D
##         R 526   0
##         D 594   0
##       Accuracy          Kappa  AccuracyLower  AccuracyUpper   AccuracyNull 
##   4.696429e-01   0.000000e+00   4.400805e-01   4.993651e-01   5.303571e-01 
## AccuracyPValue  McnemarPValue 
##   9.999790e-01  9.194240e-131 
## [1] "myfit_mdl: predict complete: 6.867000 secs"
##                          id  feats max.nTuningRuns
## 1 Random###myrandom_classfr .rnorm               0
##   min.elapsedtime.everything min.elapsedtime.final max.AUCpROC.fit
## 1                       0.29                 0.003       0.4942483
##   max.Sens.fit max.Spec.fit max.AUCROCR.fit opt.prob.threshold.fit
## 1    0.4619799    0.5265168       0.5073101                   0.55
##   max.f.score.fit max.Accuracy.fit max.AccuracyLower.fit
## 1       0.6395473        0.4700989             0.4553427
##   max.AccuracyUpper.fit max.Kappa.fit max.AUCpROC.OOB max.Sens.OOB
## 1             0.4848945             0        0.523569          0.5
##   max.Spec.OOB max.AUCROCR.OOB opt.prob.threshold.OOB max.f.score.OOB
## 1     0.547138       0.5191202                   0.55       0.6391252
##   max.Accuracy.OOB max.AccuracyLower.OOB max.AccuracyUpper.OOB
## 1        0.4696429             0.4400805             0.4993651
##   max.Kappa.OOB
## 1             0
## [1] "myfit_mdl: exit: 6.879000 secs"

# Max.cor.Y
#   Check impact of cv
#       rpart is not a good candidate since caret does not optimize cp (only tuning parameter of rpart) well
fit.models_0_chunk_df <- myadd_chunk(fit.models_0_chunk_df, 
                        paste0("fit.models_0_", "Max.cor.Y.rcv.*X*"), major.inc = FALSE,
                                    label.minor = "glmnet")

##                            label step_major step_minor      label_minor
## 3            fit.models_0_Random          1          2 myrandom_classfr
## 4 fit.models_0_Max.cor.Y.rcv.*X*          1          3           glmnet
##       bgn     end elapsed
## 3 265.981 272.872   6.891
## 4 272.872      NA      NA

ret_lst <- myfit_mdl(mdl_specs_lst = myinit_mdl_specs_lst(mdl_specs_lst = list(
    id.prefix = "Max.cor.Y.rcv.1X1", type = glb_model_type, trainControl.method = "none",
    train.method = "glmnet")),
                    indepVar = max_cor_y_x_vars, rsp_var = glb_rsp_var, 
                    fit_df = glbObsFit, OOB_df = glbObsOOB)

## [1] "myfit_mdl: enter: 0.000000 secs"
## [1] "fitting model: Max.cor.Y.rcv.1X1###glmnet"
## [1] "    indepVar: Q109244.fctr,Gender.fctr"
## [1] "myfit_mdl: setup complete: 0.695000 secs"

## Loading required package: glmnet

## Loading required package: Matrix

## 
## Attaching package: 'Matrix'

## The following object is masked from 'package:tidyr':
## 
##     expand

## Loaded glmnet 2.0-5

## Fitting alpha = 0.1, lambda = 0.00248 on full training set
## [1] "myfit_mdl: train complete: 1.532000 secs"

##             Length Class      Mode     
## a0           58    -none-     numeric  
## beta        232    dgCMatrix  S4       
## df           58    -none-     numeric  
## dim           2    -none-     numeric  
## lambda       58    -none-     numeric  
## dev.ratio    58    -none-     numeric  
## nulldev       1    -none-     numeric  
## npasses       1    -none-     numeric  
## jerr          1    -none-     numeric  
## offset        1    -none-     logical  
## classnames    2    -none-     character
## call          5    -none-     call     
## nobs          1    -none-     numeric  
## lambdaOpt     1    -none-     numeric  
## xNames        4    -none-     character
## problemType   1    -none-     character
## tuneValue     2    data.frame list     
## obsLevels     2    -none-     character
## [1] "min lambda > lambdaOpt:"
##     (Intercept)    Gender.fctrM  Q109244.fctrNo Q109244.fctrYes 
##       0.2665753      -0.2101506      -0.4308362       1.2139586 
## [1] "max lambda < lambdaOpt:"
## [1] "Feats mismatch between coefs_left & rght:"
## [1] "(Intercept)"     "Gender.fctrF"    "Gender.fctrM"    "Q109244.fctrNo" 
## [5] "Q109244.fctrYes"
## [1] "myfit_mdl: train diagnostics complete: 1.635000 secs"

##          Prediction
## Reference    R    D
##         R 1950  141
##         D 1762  595
##       Accuracy          Kappa  AccuracyLower  AccuracyUpper   AccuracyNull 
##   5.721673e-01   1.772539e-01   5.574714e-01   5.867683e-01   5.299011e-01 
## AccuracyPValue  McnemarPValue 
##   8.241814e-09  7.365212e-302

##          Prediction
## Reference   R   D
##         R 484  42
##         D 447 147
##       Accuracy          Kappa  AccuracyLower  AccuracyUpper   AccuracyNull 
##   5.633929e-01   1.605510e-01   5.337655e-01   5.926864e-01   5.303571e-01 
## AccuracyPValue  McnemarPValue 
##   1.432605e-02   1.447405e-74 
## [1] "myfit_mdl: predict complete: 6.085000 secs"
##                           id                    feats max.nTuningRuns
## 1 Max.cor.Y.rcv.1X1###glmnet Q109244.fctr,Gender.fctr               0
##   min.elapsedtime.everything min.elapsedtime.final max.AUCpROC.fit
## 1                      0.827                 0.064       0.5971118
##   max.Sens.fit max.Spec.fit max.AUCROCR.fit opt.prob.threshold.fit
## 1    0.5480631    0.6461604       0.3580613                    0.6
##   max.f.score.fit max.Accuracy.fit max.AccuracyLower.fit
## 1       0.6720662        0.5721673             0.5574714
##   max.AccuracyUpper.fit max.Kappa.fit max.AUCpROC.OOB max.Sens.OOB
## 1             0.5867683     0.1772539       0.5896897    0.5228137
##   max.Spec.OOB max.AUCROCR.OOB opt.prob.threshold.OOB max.f.score.OOB
## 1    0.6565657       0.3658672                    0.6       0.6643789
##   max.Accuracy.OOB max.AccuracyLower.OOB max.AccuracyUpper.OOB
## 1        0.5633929             0.5337655             0.5926864
##   max.Kappa.OOB
## 1      0.160551
## [1] "myfit_mdl: exit: 6.098000 secs"

if (glbMdlCheckRcv) {
    # rcv_n_folds == 1 & rcv_n_repeats > 1 crashes
    for (rcv_n_folds in seq(3, glb_rcv_n_folds + 2, 2))
        for (rcv_n_repeats in seq(1, glb_rcv_n_repeats + 2, 2)) {
            
            # Experiment specific code to avoid caret crash
    #         lcl_tune_models_df <- rbind(data.frame()
    #                             ,data.frame(method = "glmnet", parameter = "alpha", 
    #                                         vals = "0.100 0.325 0.550 0.775 1.000")
    #                             ,data.frame(method = "glmnet", parameter = "lambda",
    #                                         vals = "9.342e-02")    
    #                                     )
            
            ret_lst <- myfit_mdl(mdl_specs_lst = myinit_mdl_specs_lst(mdl_specs_lst =
                list(
                id.prefix = paste0("Max.cor.Y.rcv.", rcv_n_folds, "X", rcv_n_repeats), 
                type = glb_model_type, 
    # tune.df = lcl_tune_models_df,            
                trainControl.method = "repeatedcv",
                trainControl.number = rcv_n_folds, 
                trainControl.repeats = rcv_n_repeats,
                trainControl.classProbs = glb_is_classification,
                trainControl.summaryFunction = glbMdlMetricSummaryFn,
                train.method = "glmnet", train.metric = glbMdlMetricSummary, 
                train.maximize = glbMdlMetricMaximize)),
                                indepVar = max_cor_y_x_vars, rsp_var = glb_rsp_var, 
                                fit_df = glbObsFit, OOB_df = glbObsOOB)
        }
    # Add parallel coordinates graph of glb_models_df[, glbMdlMetricsEval] to evaluate cv parameters
    tmp_models_cols <- c("id", "max.nTuningRuns",
                        glbMdlMetricsEval[glbMdlMetricsEval %in% names(glb_models_df)],
                        grep("opt.", names(glb_models_df), fixed = TRUE, value = TRUE)) 
    print(myplot_parcoord(obs_df = subset(glb_models_df, 
                                          grepl("Max.cor.Y.rcv.", id, fixed = TRUE), 
                                            select = -feats)[, tmp_models_cols],
                          id_var = "id"))
}
        
# Useful for stacking decisions
# fit.models_0_chunk_df <- myadd_chunk(fit.models_0_chunk_df, 
#                     paste0("fit.models_0_", "Max.cor.Y[rcv.1X1.cp.0|]"), major.inc = FALSE,
#                                     label.minor = "rpart")
# 
# ret_lst <- myfit_mdl(mdl_specs_lst = myinit_mdl_specs_lst(mdl_specs_lst = list(
#     id.prefix = "Max.cor.Y.rcv.1X1.cp.0", type = glb_model_type, trainControl.method = "none",
#     train.method = "rpart",
#     tune.df=data.frame(method="rpart", parameter="cp", min=0.0, max=0.0, by=0.1))),
#                     indepVar=max_cor_y_x_vars, rsp_var=glb_rsp_var, 
#                     fit_df=glbObsFit, OOB_df=glbObsOOB)

#stop(here"); glb2Sav(); all.equal(glb_models_df, sav_models_df)
# if (glb_is_regression || glb_is_binomial) # For multinomials this model will be run next by default
ret_lst <- myfit_mdl(mdl_specs_lst = myinit_mdl_specs_lst(mdl_specs_lst = list(
                        id.prefix = "Max.cor.Y", 
                        type = glb_model_type, trainControl.method = "repeatedcv",
                        trainControl.number = glb_rcv_n_folds, 
                        trainControl.repeats = glb_rcv_n_repeats,
                        trainControl.classProbs = glb_is_classification,
                        trainControl.summaryFunction = glbMdlMetricSummaryFn,
                        trainControl.allowParallel = glbMdlAllowParallel,                        
                        train.metric = glbMdlMetricSummary, 
                        train.maximize = glbMdlMetricMaximize,    
                        train.method = "rpart")),
                    indepVar = max_cor_y_x_vars, rsp_var = glb_rsp_var, 
                    fit_df = glbObsFit, OOB_df = glbObsOOB)

## [1] "myfit_mdl: enter: 0.000000 secs"
## [1] "fitting model: Max.cor.Y##rcv#rpart"
## [1] "    indepVar: Q109244.fctr,Gender.fctr"
## [1] "myfit_mdl: setup complete: 0.722000 secs"

## Loading required package: rpart

## Aggregating results
## Selecting tuning parameters
## Fitting cp = 0.0225 on full training set
## [1] "myfit_mdl: train complete: 2.393000 secs"

## Loading required package: rpart.plot

## Call:
## rpart(formula = .outcome ~ ., control = list(minsplit = 20, minbucket = 7, 
##     cp = 0, maxcompete = 4, maxsurrogate = 5, usesurrogate = 2, 
##     surrogatestyle = 0, maxdepth = 30, xval = 0))
##   n= 4448 
## 
##           CP nsplit rel error
## 1 0.08990913      0 1.0000000
## 2 0.05930177      1 0.9100909
## 3 0.02247728      2 0.8507891
## 
## Variable importance
## Q109244.fctrYes  Q109244.fctrNo    Gender.fctrM    Gender.fctrF 
##              83              15               1               1 
## 
## Node number 1: 4448 observations,    complexity param=0.08990913
##   predicted class=D  expected loss=0.4700989  P(node) =1
##     class counts:  2091  2357
##    probabilities: 0.470 0.530 
##   left son=2 (3712 obs) right son=3 (736 obs)
##   Primary splits:
##       Q109244.fctrYes < 0.5 to the left,  improve=136.83150, (0 missing)
##       Q109244.fctrNo  < 0.5 to the right, improve= 84.31128, (0 missing)
##       Gender.fctrM    < 0.5 to the right, improve= 24.39999, (0 missing)
##       Gender.fctrF    < 0.5 to the left,  improve= 22.65952, (0 missing)
## 
## Node number 2: 3712 observations,    complexity param=0.05930177
##   predicted class=R  expected loss=0.4746767  P(node) =0.8345324
##     class counts:  1950  1762
##    probabilities: 0.525 0.475 
##   left son=4 (1980 obs) right son=5 (1732 obs)
##   Primary splits:
##       Q109244.fctrNo < 0.5 to the right, improve=24.259840, (0 missing)
##       Gender.fctrM   < 0.5 to the right, improve=10.189980, (0 missing)
##       Gender.fctrF   < 0.5 to the left,  improve= 8.193561, (0 missing)
##   Surrogate splits:
##       Gender.fctrM < 0.5 to the right, agree=0.571, adj=0.080, (0 split)
##       Gender.fctrF < 0.5 to the left,  agree=0.563, adj=0.063, (0 split)
## 
## Node number 3: 736 observations
##   predicted class=D  expected loss=0.1915761  P(node) =0.1654676
##     class counts:   141   595
##    probabilities: 0.192 0.808 
## 
## Node number 4: 1980 observations
##   predicted class=R  expected loss=0.4212121  P(node) =0.4451439
##     class counts:  1146   834
##    probabilities: 0.579 0.421 
## 
## Node number 5: 1732 observations
##   predicted class=D  expected loss=0.4642032  P(node) =0.3893885
##     class counts:   804   928
##    probabilities: 0.464 0.536 
## 
## n= 4448 
## 
## node), split, n, loss, yval, (yprob)
##       * denotes terminal node
## 
## 1) root 4448 2091 D (0.4700989 0.5299011)  
##   2) Q109244.fctrYes< 0.5 3712 1762 R (0.5253233 0.4746767)  
##     4) Q109244.fctrNo>=0.5 1980  834 R (0.5787879 0.4212121) *
##     5) Q109244.fctrNo< 0.5 1732  804 D (0.4642032 0.5357968) *
##   3) Q109244.fctrYes>=0.5 736  141 D (0.1915761 0.8084239) *
## [1] "myfit_mdl: train diagnostics complete: 4.350000 secs"

##          Prediction
## Reference    R    D
##         R 1950  141
##         D 1762  595
##       Accuracy          Kappa  AccuracyLower  AccuracyUpper   AccuracyNull 
##   5.721673e-01   1.772539e-01   5.574714e-01   5.867683e-01   5.299011e-01 
## AccuracyPValue  McnemarPValue 
##   8.241814e-09  7.365212e-302

##          Prediction
## Reference   R   D
##         R 484  42
##         D 447 147
##       Accuracy          Kappa  AccuracyLower  AccuracyUpper   AccuracyNull 
##   5.633929e-01   1.605510e-01   5.337655e-01   5.926864e-01   5.303571e-01 
## AccuracyPValue  McnemarPValue 
##   1.432605e-02   1.447405e-74 
## [1] "myfit_mdl: predict complete: 13.460000 secs"
##                     id                    feats max.nTuningRuns
## 1 Max.cor.Y##rcv#rpart Q109244.fctr,Gender.fctr               5
##   min.elapsedtime.everything min.elapsedtime.final max.AUCpROC.fit
## 1                       1.66                  0.02       0.5971118
##   max.Sens.fit max.Spec.fit max.AUCROCR.fit opt.prob.threshold.fit
## 1    0.5480631    0.6461604       0.3676308                   0.55
##   max.f.score.fit max.Accuracy.fit max.AccuracyLower.fit
## 1       0.6720662         0.600045             0.5574714
##   max.AccuracyUpper.fit max.Kappa.fit max.AUCpROC.OOB max.Sens.OOB
## 1             0.5867683     0.1947896       0.5896897    0.5228137
##   max.Spec.OOB max.AUCROCR.OOB opt.prob.threshold.OOB max.f.score.OOB
## 1    0.6565657       0.3774772                   0.55       0.6643789
##   max.Accuracy.OOB max.AccuracyLower.OOB max.AccuracyUpper.OOB
## 1        0.5633929             0.5337655             0.5926864
##   max.Kappa.OOB max.AccuracySD.fit max.KappaSD.fit
## 1      0.160551          0.0124035      0.02559319
## [1] "myfit_mdl: exit: 13.476000 secs"

if ((length(glbFeatsDateTime) > 0) && 
    (sum(grepl(paste(names(glbFeatsDateTime), "\\.day\\.minutes\\.poly\\.", sep = ""),
               names(glbObsAll))) > 0)) {
    fit.models_0_chunk_df <- myadd_chunk(fit.models_0_chunk_df, 
                    paste0("fit.models_0_", "Max.cor.Y.Time.Poly"), major.inc = FALSE,
                                    label.minor = "glmnet")

    indepVars <- c(max_cor_y_x_vars, 
            grep(paste(names(glbFeatsDateTime), "\\.day\\.minutes\\.poly\\.", sep = ""),
                        names(glbObsAll), value = TRUE))
    indepVars <- myadjustInteractionFeats(glb_feats_df, indepVars)
    ret_lst <- myfit_mdl(mdl_specs_lst = myinit_mdl_specs_lst(mdl_specs_lst = list(
            id.prefix = "Max.cor.Y.Time.Poly", 
            type = glb_model_type, trainControl.method = "repeatedcv",
            trainControl.number = glb_rcv_n_folds, trainControl.repeats = glb_rcv_n_repeats,
            trainControl.classProbs = glb_is_classification,
            trainControl.summaryFunction = glbMdlMetricSummaryFn,
            trainControl.allowParallel = glbMdlAllowParallel,            
            train.metric = glbMdlMetricSummary, 
            train.maximize = glbMdlMetricMaximize,    
            train.method = "glmnet")),
        indepVar = indepVars,
        rsp_var = glb_rsp_var, 
        fit_df = glbObsFit, OOB_df = glbObsOOB)
}

if ((length(glbFeatsDateTime) > 0) && 
    (sum(grepl(paste(names(glbFeatsDateTime), "\\.last[[:digit:]]", sep = ""),
               names(glbObsAll))) > 0)) {
    fit.models_0_chunk_df <- myadd_chunk(fit.models_0_chunk_df, 
                    paste0("fit.models_0_", "Max.cor.Y.Time.Lag"), major.inc = FALSE,
                                    label.minor = "glmnet")

    indepVars <- c(max_cor_y_x_vars, 
            grep(paste(names(glbFeatsDateTime), "\\.last[[:digit:]]", sep = ""),
                        names(glbObsAll), value = TRUE))
    indepVars <- myadjustInteractionFeats(glb_feats_df, indepVars)
    ret_lst <- myfit_mdl(mdl_specs_lst = myinit_mdl_specs_lst(mdl_specs_lst = list(
        id.prefix = "Max.cor.Y.Time.Lag", 
        type = glb_model_type, 
        tune.df = glbMdlTuneParams,        
        trainControl.method = "repeatedcv",
        trainControl.number = glb_rcv_n_folds, trainControl.repeats = glb_rcv_n_repeats,
        trainControl.classProbs = glb_is_classification,
        trainControl.summaryFunction = glbMdlMetricSummaryFn,
        trainControl.allowParallel = glbMdlAllowParallel,        
        train.metric = glbMdlMetricSummary, 
        train.maximize = glbMdlMetricMaximize,    
        train.method = "glmnet")),
        indepVar = indepVars,
        rsp_var = glb_rsp_var, 
        fit_df = glbObsFit, OOB_df = glbObsOOB)
}

if (length(glbFeatsText) > 0) {
    fit.models_0_chunk_df <- myadd_chunk(fit.models_0_chunk_df, 
                    paste0("fit.models_0_", "Txt.*"), major.inc = FALSE,
                                    label.minor = "glmnet")

    indepVars <- c(max_cor_y_x_vars)
    for (txtFeat in names(glbFeatsText))
        indepVars <- union(indepVars, 
            grep(paste(str_to_upper(substr(txtFeat, 1, 1)), "\\.(?!([T|P]\\.))", sep = ""),
                        names(glbObsAll), perl = TRUE, value = TRUE))
    indepVars <- myadjustInteractionFeats(glb_feats_df, indepVars)
    ret_lst <- myfit_mdl(mdl_specs_lst = myinit_mdl_specs_lst(mdl_specs_lst = list(
        id.prefix = "Max.cor.Y.Text.nonTP", 
        type = glb_model_type, 
        tune.df = glbMdlTuneParams,        
        trainControl.method = "repeatedcv",
        trainControl.number = glb_rcv_n_folds, trainControl.repeats = glb_rcv_n_repeats,
        trainControl.classProbs = glb_is_classification,
        trainControl.summaryFunction = glbMdlMetricSummaryFn,
        trainControl.allowParallel = glbMdlAllowParallel,                                
        train.metric = glbMdlMetricSummary, 
        train.maximize = glbMdlMetricMaximize,    
        train.method = "glmnet")),
        indepVar = indepVars,
        rsp_var = glb_rsp_var, 
        fit_df = glbObsFit, OOB_df = glbObsOOB)

    indepVars <- c(max_cor_y_x_vars)
    for (txtFeat in names(glbFeatsText))
        indepVars <- union(indepVars, 
            grep(paste(str_to_upper(substr(txtFeat, 1, 1)), "\\.T\\.", sep = ""),
                        names(glbObsAll), perl = TRUE, value = TRUE))
    indepVars <- myadjustInteractionFeats(glb_feats_df, indepVars)
    ret_lst <- myfit_mdl(mdl_specs_lst = myinit_mdl_specs_lst(mdl_specs_lst = list(
        id.prefix = "Max.cor.Y.Text.onlyT", 
        type = glb_model_type, 
        tune.df = glbMdlTuneParams,        
        trainControl.method = "repeatedcv",
        trainControl.number = glb_rcv_n_folds, trainControl.repeats = glb_rcv_n_repeats,
        trainControl.classProbs = glb_is_classification,
        trainControl.summaryFunction = glbMdlMetricSummaryFn,
        train.metric = glbMdlMetricSummary, 
        train.maximize = glbMdlMetricMaximize,    
        train.method = "glmnet")),
        indepVar = indepVars,
        rsp_var = glb_rsp_var, 
        fit_df = glbObsFit, OOB_df = glbObsOOB)

    indepVars <- c(max_cor_y_x_vars)
    for (txtFeat in names(glbFeatsText))
        indepVars <- union(indepVars, 
            grep(paste(str_to_upper(substr(txtFeat, 1, 1)), "\\.P\\.", sep = ""),
                        names(glbObsAll), perl = TRUE, value = TRUE))
    indepVars <- myadjustInteractionFeats(glb_feats_df, indepVars)
    ret_lst <- myfit_mdl(mdl_specs_lst = myinit_mdl_specs_lst(mdl_specs_lst = list(
        id.prefix = "Max.cor.Y.Text.onlyP", 
        type = glb_model_type, 
        tune.df = glbMdlTuneParams,        
        trainControl.method = "repeatedcv",
        trainControl.number = glb_rcv_n_folds, trainControl.repeats = glb_rcv_n_repeats,
        trainControl.classProbs = glb_is_classification,
        trainControl.summaryFunction = glbMdlMetricSummaryFn,
        trainControl.allowParallel = glbMdlAllowParallel,        
        train.metric = glbMdlMetricSummary, 
        train.maximize = glbMdlMetricMaximize,    
        train.method = "glmnet")),
        indepVar = indepVars,
        rsp_var = glb_rsp_var, 
        fit_df = glbObsFit, OOB_df = glbObsOOB)
}

# Interactions.High.cor.Y
if (length(int_feats <- setdiff(setdiff(unique(glb_feats_df$cor.high.X), NA), 
                                subset(glb_feats_df, nzv)$id)) > 0) {
    fit.models_0_chunk_df <- myadd_chunk(fit.models_0_chunk_df, 
                    paste0("fit.models_0_", "Interact.High.cor.Y"), major.inc = FALSE,
                                    label.minor = "glmnet")

    ret_lst <- myfit_mdl(mdl_specs_lst=myinit_mdl_specs_lst(mdl_specs_lst=list(
        id.prefix="Interact.High.cor.Y", 
        type=glb_model_type, trainControl.method="repeatedcv",
        trainControl.number=glb_rcv_n_folds, trainControl.repeats=glb_rcv_n_repeats,
        trainControl.classProbs = glb_is_classification,
        trainControl.summaryFunction = glbMdlMetricSummaryFn,
        trainControl.allowParallel = glbMdlAllowParallel,
        train.metric = glbMdlMetricSummary, 
        train.maximize = glbMdlMetricMaximize,    
        train.method="glmnet")),
        indepVar=c(max_cor_y_x_vars, paste(max_cor_y_x_vars[1], int_feats, sep=":")),
        rsp_var=glb_rsp_var, 
        fit_df=glbObsFit, OOB_df=glbObsOOB)
}

##                              label step_major step_minor label_minor
## 4   fit.models_0_Max.cor.Y.rcv.*X*          1          3      glmnet
## 5 fit.models_0_Interact.High.cor.Y          1          4      glmnet
##       bgn     end elapsed
## 4 272.872 292.488  19.616
## 5 292.488      NA      NA
## [1] "myfit_mdl: enter: 0.000000 secs"
## [1] "fitting model: Interact.High.cor.Y##rcv#glmnet"
## [1] "    indepVar: Q109244.fctr,Gender.fctr,Q109244.fctr:Q99480.fctr,Q109244.fctr:Q98078.fctr,Q109244.fctr:Q100689.fctr,Q109244.fctr:Q108855.fctr,Q109244.fctr:Q123621.fctr,Q109244.fctr:Q122771.fctr,Q109244.fctr:Q120472.fctr,Q109244.fctr:Q106272.fctr"
## [1] "myfit_mdl: setup complete: 0.725000 secs"
## Aggregating results
## Selecting tuning parameters
## Fitting alpha = 1, lambda = 0.00248 on full training set
## [1] "myfit_mdl: train complete: 6.038000 secs"

## Warning in myfit_mdl(mdl_specs_lst = myinit_mdl_specs_lst(mdl_specs_lst
## = list(id.prefix = "Interact.High.cor.Y", : model's bestTune found at an
## extreme of tuneGrid for parameter: alpha

##             Length Class      Mode     
## a0            69   -none-     numeric  
## beta        3588   dgCMatrix  S4       
## df            69   -none-     numeric  
## dim            2   -none-     numeric  
## lambda        69   -none-     numeric  
## dev.ratio     69   -none-     numeric  
## nulldev        1   -none-     numeric  
## npasses        1   -none-     numeric  
## jerr           1   -none-     numeric  
## offset         1   -none-     logical  
## classnames     2   -none-     character
## call           5   -none-     call     
## nobs           1   -none-     numeric  
## lambdaOpt      1   -none-     numeric  
## xNames        52   -none-     character
## problemType    1   -none-     character
## tuneValue      2   data.frame list     
## obsLevels      2   -none-     character
## [1] "min lambda > lambdaOpt:"
##                        (Intercept)                       Gender.fctrM 
##                        0.214462392                       -0.145896015 
##                     Q109244.fctrNo                    Q109244.fctrYes 
##                       -0.216275985                        0.903504361 
##      Q109244.fctrNA:Q100689.fctrNo     Q109244.fctrYes:Q100689.fctrNo 
##                        0.208349081                       -0.004870424 
##     Q109244.fctrNA:Q100689.fctrYes     Q109244.fctrNo:Q100689.fctrYes 
##                        0.397412088                        0.048879968 
##    Q109244.fctrYes:Q100689.fctrYes      Q109244.fctrNA:Q106272.fctrNo 
##                        0.262128025                        0.066457216 
##      Q109244.fctrNo:Q106272.fctrNo     Q109244.fctrYes:Q106272.fctrNo 
##                        0.074008287                       -0.069508937 
##     Q109244.fctrNA:Q106272.fctrYes     Q109244.fctrNo:Q106272.fctrYes 
##                       -0.125437842                       -0.155157463 
##    Q109244.fctrYes:Q106272.fctrYes  Q109244.fctrNA:Q108855.fctrUmm... 
##                        0.019996659                       -0.325540110 
##  Q109244.fctrNo:Q108855.fctrUmm... Q109244.fctrYes:Q108855.fctrUmm... 
##                        0.065090912                        0.025966636 
##    Q109244.fctrNo:Q108855.fctrYes!     Q109244.fctrNA:Q120472.fctrArt 
##                       -0.166309419                        0.045175676 
##    Q109244.fctrYes:Q120472.fctrArt Q109244.fctrNA:Q120472.fctrScience 
##                        0.050613747                       -0.081869305 
## Q109244.fctrNo:Q120472.fctrScience      Q109244.fctrNA:Q122771.fctrPc 
##                       -0.045206278                        0.026381445 
##      Q109244.fctrNo:Q122771.fctrPc      Q109244.fctrNA:Q122771.fctrPt 
##                       -0.079571797                       -0.056075353 
##      Q109244.fctrNo:Q122771.fctrPt     Q109244.fctrYes:Q122771.fctrPt 
##                       -0.312370610                       -0.184990100 
##      Q109244.fctrNA:Q123621.fctrNo     Q109244.fctrYes:Q123621.fctrNo 
##                       -0.055648138                        0.250541256 
##     Q109244.fctrNo:Q123621.fctrYes    Q109244.fctrYes:Q123621.fctrYes 
##                       -0.118706388                        0.234767924 
##       Q109244.fctrNA:Q98078.fctrNo       Q109244.fctrNo:Q98078.fctrNo 
##                        0.041213383                        0.052285528 
##      Q109244.fctrNA:Q98078.fctrYes     Q109244.fctrYes:Q98078.fctrYes 
##                        0.124665272                        0.101993831 
##       Q109244.fctrNA:Q99480.fctrNo       Q109244.fctrNo:Q99480.fctrNo 
##                        0.285510277                        0.345084748 
##      Q109244.fctrYes:Q99480.fctrNo      Q109244.fctrNA:Q99480.fctrYes 
##                        0.054445838                       -0.272288871 
## [1] "max lambda < lambdaOpt:"
##                        (Intercept)                       Gender.fctrM 
##                        0.215265192                       -0.146631129 
##                     Q109244.fctrNo                    Q109244.fctrYes 
##                       -0.232130198                        0.919519640 
##      Q109244.fctrNA:Q100689.fctrNo     Q109244.fctrYes:Q100689.fctrNo 
##                        0.218265269                       -0.019284614 
##     Q109244.fctrNA:Q100689.fctrYes     Q109244.fctrNo:Q100689.fctrYes 
##                        0.407359039                        0.052982570 
##    Q109244.fctrYes:Q100689.fctrYes      Q109244.fctrNA:Q106272.fctrNo 
##                        0.257862986                        0.066427041 
##      Q109244.fctrNo:Q106272.fctrNo     Q109244.fctrYes:Q106272.fctrNo 
##                        0.075834582                       -0.093768371 
##     Q109244.fctrNA:Q106272.fctrYes     Q109244.fctrNo:Q106272.fctrYes 
##                       -0.132132143                       -0.157097270 
##    Q109244.fctrYes:Q106272.fctrYes  Q109244.fctrNA:Q108855.fctrUmm... 
##                        0.006954145                       -0.334091989 
##  Q109244.fctrNo:Q108855.fctrUmm... Q109244.fctrYes:Q108855.fctrUmm... 
##                        0.073183683                        0.029348178 
##    Q109244.fctrNo:Q108855.fctrYes!     Q109244.fctrNA:Q120472.fctrArt 
##                       -0.161489197                        0.047013857 
##    Q109244.fctrYes:Q120472.fctrArt Q109244.fctrNA:Q120472.fctrScience 
##                        0.053722253                       -0.085734575 
## Q109244.fctrNo:Q120472.fctrScience      Q109244.fctrNA:Q122771.fctrPc 
##                       -0.046328701                        0.031858306 
##      Q109244.fctrNo:Q122771.fctrPc      Q109244.fctrNA:Q122771.fctrPt 
##                       -0.086494530                       -0.058868145 
##      Q109244.fctrNo:Q122771.fctrPt     Q109244.fctrYes:Q122771.fctrPt 
##                       -0.320607007                       -0.200384753 
##      Q109244.fctrNA:Q123621.fctrNo     Q109244.fctrYes:Q123621.fctrNo 
##                       -0.062593721                        0.268705396 
##     Q109244.fctrNo:Q123621.fctrYes    Q109244.fctrYes:Q123621.fctrYes 
##                       -0.120160110                        0.251231071 
##       Q109244.fctrNA:Q98078.fctrNo       Q109244.fctrNo:Q98078.fctrNo 
##                        0.050653191                        0.069505105 
##      Q109244.fctrNA:Q98078.fctrYes      Q109244.fctrNo:Q98078.fctrYes 
##                        0.134149587                        0.017836961 
##     Q109244.fctrYes:Q98078.fctrYes       Q109244.fctrNA:Q99480.fctrNo 
##                        0.109598128                        0.281737023 
##       Q109244.fctrNo:Q99480.fctrNo      Q109244.fctrYes:Q99480.fctrNo 
##                        0.348068744                        0.053827527 
##      Q109244.fctrNA:Q99480.fctrYes     Q109244.fctrYes:Q99480.fctrYes 
##                       -0.284681230                       -0.012138329 
## [1] "myfit_mdl: train diagnostics complete: 6.810000 secs"

##          Prediction
## Reference    R    D
##         R 1929  162
##         D 1715  642
##       Accuracy          Kappa  AccuracyLower  AccuracyUpper   AccuracyNull 
##   5.780126e-01   1.870655e-01   5.633390e-01   5.925837e-01   5.299011e-01 
## AccuracyPValue  McnemarPValue 
##   6.342747e-11  4.877355e-281

##          Prediction
## Reference   R   D
##         R 481  45
##         D 433 161
##       Accuracy          Kappa  AccuracyLower  AccuracyUpper   AccuracyNull 
##   5.732143e-01   1.779779e-01   5.436402e-01   6.024028e-01   5.303571e-01 
## AccuracyPValue  McnemarPValue 
##   2.187684e-03   4.121616e-70 
## [1] "myfit_mdl: predict complete: 12.643000 secs"
##                                id
## 1 Interact.High.cor.Y##rcv#glmnet
##                                                                                                                                                                                                                                    feats
## 1 Q109244.fctr,Gender.fctr,Q109244.fctr:Q99480.fctr,Q109244.fctr:Q98078.fctr,Q109244.fctr:Q100689.fctr,Q109244.fctr:Q108855.fctr,Q109244.fctr:Q123621.fctr,Q109244.fctr:Q122771.fctr,Q109244.fctr:Q120472.fctr,Q109244.fctr:Q106272.fctr
##   max.nTuningRuns min.elapsedtime.everything min.elapsedtime.final
## 1              25                      5.291                 0.358
##   max.AUCpROC.fit max.Sens.fit max.Spec.fit max.AUCROCR.fit
## 1       0.6184781    0.5958871    0.6410692       0.3319465
##   opt.prob.threshold.fit max.f.score.fit max.Accuracy.fit
## 1                   0.65       0.6727114        0.6058167
##   max.AccuracyLower.fit max.AccuracyUpper.fit max.Kappa.fit
## 1              0.563339             0.5925837     0.2088694
##   max.AUCpROC.OOB max.Sens.OOB max.Spec.OOB max.AUCROCR.OOB
## 1       0.6031353    0.5665399    0.6397306       0.3571392
##   opt.prob.threshold.OOB max.f.score.OOB max.Accuracy.OOB
## 1                   0.65       0.6680556        0.5732143
##   max.AccuracyLower.OOB max.AccuracyUpper.OOB max.Kappa.OOB
## 1             0.5436402             0.6024028     0.1779779
##   max.AccuracySD.fit max.KappaSD.fit
## 1          0.0131213      0.02732571
## [1] "myfit_mdl: exit: 12.658000 secs"

# Low.cor.X
fit.models_0_chunk_df <- myadd_chunk(fit.models_0_chunk_df, 
                        paste0("fit.models_0_", "Low.cor.X"), major.inc = FALSE,
                                     label.minor = "glmnet")

##                              label step_major step_minor label_minor
## 5 fit.models_0_Interact.High.cor.Y          1          4      glmnet
## 6           fit.models_0_Low.cor.X          1          5      glmnet
##       bgn     end elapsed
## 5 292.488 305.178   12.69
## 6 305.178      NA      NA

indepVar <- mygetIndepVar(glb_feats_df)
ret_lst <- myfit_mdl(mdl_specs_lst = myinit_mdl_specs_lst(mdl_specs_lst = list(
            id.prefix = "Low.cor.X", 
            type = glb_model_type, 
            tune.df = glbMdlTuneParams,        
            trainControl.method = "repeatedcv",
            trainControl.number = glb_rcv_n_folds, trainControl.repeats = glb_rcv_n_repeats,
            trainControl.classProbs = glb_is_classification,
            trainControl.summaryFunction = glbMdlMetricSummaryFn,
            trainControl.allowParallel = glbMdlAllowParallel,
            train.metric = glbMdlMetricSummary, 
            train.maximize = glbMdlMetricMaximize,    
            train.method = "glmnet")),
        indepVar = indepVar, rsp_var = glb_rsp_var, 
        fit_df = glbObsFit, OOB_df = glbObsOOB)

## [1] "myfit_mdl: enter: 0.000000 secs"
## [1] "fitting model: Low.cor.X##rcv#glmnet"
## [1] "    indepVar: Q109244.fctr,Hhold.fctr,Edn.fctr,Q101163.fctr,Q100689.fctr,Q98078.fctr,Q99716.fctr,Q120379.fctr,Q121699.fctr,Q105840.fctr,Q113583.fctr,Q115195.fctr,Q102089.fctr,Q98059.fctr,Q114386.fctr,Q100680.fctr,Q108342.fctr,Q111848.fctr,YOB.Age.fctr,Q118892.fctr,Q102687.fctr,Q115390.fctr,Q119851.fctr,Q114517.fctr,Q120012.fctr,Q109367.fctr,Q114961.fctr,Q121700.fctr,Q124122.fctr,Q111220.fctr,Q113992.fctr,Q121011.fctr,Q106042.fctr,Q116448.fctr,Q116601.fctr,Q104996.fctr,Q102906.fctr,Q113584.fctr,Q108950.fctr,Q102674.fctr,Q103293.fctr,Q112478.fctr,Q114748.fctr,Q107491.fctr,Q100562.fctr,Q108617.fctr,Q100010.fctr,Q115602.fctr,Q116953.fctr,Q115610.fctr,Q106997.fctr,Q120978.fctr,Q112512.fctr,Q108343.fctr,Q96024.fctr,Q106389.fctr,.rnorm,Q108754.fctr,Q98578.fctr,Q101162.fctr,Q115777.fctr,Q99581.fctr,Q124742.fctr,Q116797.fctr,Q112270.fctr,Q118237.fctr,Q119650.fctr,Q111580.fctr,Q123464.fctr,Q117193.fctr,Q99982.fctr,Q108856.fctr,Q118233.fctr,Q102289.fctr,Q116197.fctr,Income.fctr,Q118232.fctr,Q120194.fctr,Q114152.fctr,Q122770.fctr,Q117186.fctr,Q105655.fctr,Q106993.fctr,Q119334.fctr,Q122120.fctr,Q116441.fctr,Q118117.fctr,Q123621.fctr,Q122769.fctr,Q120650.fctr,Q98869.fctr,Q107869.fctr,Q120014.fctr,Q115899.fctr,Q106388.fctr,Q99480.fctr,Q122771.fctr,Q108855.fctr,Q110740.fctr,Q106272.fctr,Q101596.fctr,Q116881.fctr,Q120472.fctr,Q98197.fctr,Q113181.fctr,Q115611.fctr,Gender.fctr,Hhold.fctr:.clusterid.fctr"
## [1] "myfit_mdl: setup complete: 0.707000 secs"
## Aggregating results
## Selecting tuning parameters
## Fitting alpha = 0.325, lambda = 0.0534 on full training set
## [1] "myfit_mdl: train complete: 25.220000 secs"

## Warning in myfit_mdl(mdl_specs_lst = myinit_mdl_specs_lst(mdl_specs_lst
## = list(id.prefix = "Low.cor.X", : model's bestTune found at an extreme of
## tuneGrid for parameter: lambda

##             Length Class      Mode     
## a0             78  -none-     numeric  
## beta        19734  dgCMatrix  S4       
## df             78  -none-     numeric  
## dim             2  -none-     numeric  
## lambda         78  -none-     numeric  
## dev.ratio      78  -none-     numeric  
## nulldev         1  -none-     numeric  
## npasses         1  -none-     numeric  
## jerr            1  -none-     numeric  
## offset          1  -none-     logical  
## classnames      2  -none-     character
## call            5  -none-     call     
## nobs            1  -none-     numeric  
## lambdaOpt       1  -none-     numeric  
## xNames        253  -none-     character
## problemType     1  -none-     character
## tuneValue       2  data.frame list     
## obsLevels       2  -none-     character
## [1] "min lambda > lambdaOpt:"
##       (Intercept)      Gender.fctrM     Hhold.fctrMKy     Hhold.fctrPKn 
##      0.1759341202     -0.0577396529     -0.0655451865      0.3282606962 
##   Q101163.fctrDad   Q101163.fctrMom    Q109244.fctrNo   Q109244.fctrYes 
##     -0.0462347606      0.0361283260     -0.3170070971      0.7976156342 
##    Q110740.fctrPC    Q113181.fctrNo   Q113181.fctrYes    Q115611.fctrNo 
##     -0.0120813108      0.0348921948     -0.0798654082      0.0770473552 
##   Q115611.fctrYes Q116881.fctrHappy Q116881.fctrRight    Q118232.fctrId 
##     -0.3105852699      0.0087382029     -0.0814535814      0.0180660706 
##    Q119851.fctrNo    Q120379.fctrNo   Q120379.fctrYes     Q98197.fctrNo 
##     -0.0297897358     -0.0006993182      0.0272478036      0.1639383807 
##    Q98197.fctrYes     Q98869.fctrNo     Q99480.fctrNo 
##     -0.0341915807      0.1609634790      0.0576623212 
## [1] "max lambda < lambdaOpt:"
##       (Intercept)      Gender.fctrM     Hhold.fctrMKy     Hhold.fctrPKn 
##       0.180590790      -0.063356510      -0.071963008       0.358397790 
##     Income.fctr.Q   Q101163.fctrDad   Q101163.fctrMom    Q109244.fctrNo 
##      -0.003003778      -0.055383160       0.041352778      -0.326547933 
##   Q109244.fctrYes    Q110740.fctrPC    Q113181.fctrNo   Q113181.fctrYes 
##       0.811987955      -0.023695743       0.041178168      -0.082431818 
##    Q115611.fctrNo   Q115611.fctrYes    Q115899.fctrCs Q116881.fctrHappy 
##       0.080608650      -0.318215302       0.001114558       0.019171363 
## Q116881.fctrRight    Q118232.fctrId    Q119851.fctrNo    Q120379.fctrNo 
##      -0.090653797       0.032794176      -0.043521175      -0.004314002 
##   Q120379.fctrYes    Q121699.fctrNo    Q122771.fctrPt     Q98197.fctrNo 
##       0.038379682      -0.010439127      -0.000907088       0.171596860 
##    Q98197.fctrYes     Q98869.fctrNo     Q99480.fctrNo 
##      -0.034226771       0.174722108       0.068918234 
## [1] "myfit_mdl: train diagnostics complete: 25.938000 secs"

##          Prediction
## Reference    R    D
##         R 1911  180
##         D 1655  702
##       Accuracy          Kappa  AccuracyLower  AccuracyUpper   AccuracyNull 
##   5.874550e-01   2.036470e-01   5.728218e-01   6.019734e-01   5.299011e-01 
## AccuracyPValue  McnemarPValue 
##   6.718265e-15  1.812597e-259

##          Prediction
## Reference   R   D
##         R 494  32
##         D 467 127
##       Accuracy          Kappa  AccuracyLower  AccuracyUpper   AccuracyNull 
##   5.544643e-01   1.460545e-01   5.247985e-01   5.838432e-01   5.303571e-01 
## AccuracyPValue  McnemarPValue 
##   5.617624e-02   4.430516e-84 
## [1] "myfit_mdl: predict complete: 36.299000 secs"
##                      id
## 1 Low.cor.X##rcv#glmnet
##                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               feats
## 1 Q109244.fctr,Hhold.fctr,Edn.fctr,Q101163.fctr,Q100689.fctr,Q98078.fctr,Q99716.fctr,Q120379.fctr,Q121699.fctr,Q105840.fctr,Q113583.fctr,Q115195.fctr,Q102089.fctr,Q98059.fctr,Q114386.fctr,Q100680.fctr,Q108342.fctr,Q111848.fctr,YOB.Age.fctr,Q118892.fctr,Q102687.fctr,Q115390.fctr,Q119851.fctr,Q114517.fctr,Q120012.fctr,Q109367.fctr,Q114961.fctr,Q121700.fctr,Q124122.fctr,Q111220.fctr,Q113992.fctr,Q121011.fctr,Q106042.fctr,Q116448.fctr,Q116601.fctr,Q104996.fctr,Q102906.fctr,Q113584.fctr,Q108950.fctr,Q102674.fctr,Q103293.fctr,Q112478.fctr,Q114748.fctr,Q107491.fctr,Q100562.fctr,Q108617.fctr,Q100010.fctr,Q115602.fctr,Q116953.fctr,Q115610.fctr,Q106997.fctr,Q120978.fctr,Q112512.fctr,Q108343.fctr,Q96024.fctr,Q106389.fctr,.rnorm,Q108754.fctr,Q98578.fctr,Q101162.fctr,Q115777.fctr,Q99581.fctr,Q124742.fctr,Q116797.fctr,Q112270.fctr,Q118237.fctr,Q119650.fctr,Q111580.fctr,Q123464.fctr,Q117193.fctr,Q99982.fctr,Q108856.fctr,Q118233.fctr,Q102289.fctr,Q116197.fctr,Income.fctr,Q118232.fctr,Q120194.fctr,Q114152.fctr,Q122770.fctr,Q117186.fctr,Q105655.fctr,Q106993.fctr,Q119334.fctr,Q122120.fctr,Q116441.fctr,Q118117.fctr,Q123621.fctr,Q122769.fctr,Q120650.fctr,Q98869.fctr,Q107869.fctr,Q120014.fctr,Q115899.fctr,Q106388.fctr,Q99480.fctr,Q122771.fctr,Q108855.fctr,Q110740.fctr,Q106272.fctr,Q101596.fctr,Q116881.fctr,Q120472.fctr,Q98197.fctr,Q113181.fctr,Q115611.fctr,Gender.fctr,Hhold.fctr:.clusterid.fctr
##   max.nTuningRuns min.elapsedtime.everything min.elapsedtime.final
## 1              25                     24.392                 2.084
##   max.AUCpROC.fit max.Sens.fit max.Spec.fit max.AUCROCR.fit
## 1       0.6273089    0.5155428    0.7390751       0.3048695
##   opt.prob.threshold.fit max.f.score.fit max.Accuracy.fit
## 1                    0.6       0.6756231        0.6241754
##   max.AccuracyLower.fit max.AccuracyUpper.fit max.Kappa.fit
## 1             0.5728218             0.6019734     0.2388455
##   max.AUCpROC.OOB max.Sens.OOB max.Spec.OOB max.AUCROCR.OOB
## 1       0.6237374          0.5    0.7474747       0.3184875
##   opt.prob.threshold.OOB max.f.score.OOB max.Accuracy.OOB
## 1                    0.7        0.664425        0.5544643
##   max.AccuracyLower.OOB max.AccuracyUpper.OOB max.Kappa.OOB
## 1             0.5247985             0.5838432     0.1460545
##   max.AccuracySD.fit max.KappaSD.fit
## 1        0.004938833      0.01189291
## [1] "myfit_mdl: exit: 36.314000 secs"

fit.models_0_chunk_df <- 
    myadd_chunk(fit.models_0_chunk_df, "fit.models_0_end", major.inc = FALSE,
                label.minor = "teardown")

##                    label step_major step_minor label_minor     bgn     end
## 6 fit.models_0_Low.cor.X          1          5      glmnet 305.178 341.545
## 7       fit.models_0_end          1          6    teardown 341.545      NA
##   elapsed
## 6  36.367
## 7      NA

rm(ret_lst)

glb_chunks_df <- myadd_chunk(glb_chunks_df, "fit.models", major.inc = FALSE)

##        label step_major step_minor label_minor     bgn     end elapsed
## 4 fit.models          4          0           0 259.549 341.558  82.009
## 5 fit.models          4          1           1 341.558      NA      NA

fit.models_1_chunk_df <- myadd_chunk(NULL, "fit.models_1_bgn", label.minor = "setup")

##              label step_major step_minor label_minor     bgn end elapsed
## 1 fit.models_1_bgn          1          0       setup 345.908  NA      NA

# refactor code for outliers / ensure all model runs exclude outliers in this chunk ???

#stop(here"); glb2Sav(); all.equal(glb_models_df, sav_models_df)
topindep_var <- NULL; interact_vars <- NULL;
for (mdl_id_pfx in names(glbMdlFamilies)) {
    fit.models_1_chunk_df <- 
        myadd_chunk(fit.models_1_chunk_df, paste0("fit.models_1_", mdl_id_pfx),
                    major.inc = FALSE, label.minor = "setup")

    indepVar <- NULL;

    if (grepl("\\.Interact", mdl_id_pfx)) {
        if (is.null(topindep_var) && is.null(interact_vars)) {
        #   select best glmnet model upto now
            dsp_models_df <- orderBy(model_sel_frmla <- get_model_sel_frmla(),
                                     glb_models_df)
            dsp_models_df <- subset(dsp_models_df, 
                                    grepl(".glmnet", id, fixed = TRUE))
            bst_mdl_id <- dsp_models_df$id[1]
            mdl_id_pfx <- 
                paste(c(head(unlist(strsplit(bst_mdl_id, "[.]")), -1), "Interact"),
                      collapse=".")
        #   select important features
            if (is.null(bst_featsimp_df <- 
                        myget_feats_importance(glb_models_lst[[bst_mdl_id]]))) {
                warning("Base model for RFE.Interact: ", bst_mdl_id, 
                        " has no important features")
                next
            }    
            
            topindep_ix <- 1
            while (is.null(topindep_var) && (topindep_ix <= nrow(bst_featsimp_df))) {
                topindep_var <- row.names(bst_featsimp_df)[topindep_ix]
                if (grepl(".fctr", topindep_var, fixed=TRUE))
                    topindep_var <- 
                        paste0(unlist(strsplit(topindep_var, ".fctr"))[1], ".fctr")
                if (topindep_var %in% names(glbFeatsInteractionOnly)) {
                    topindep_var <- NULL; topindep_ix <- topindep_ix + 1
                } else break
            }
            
        #   select features with importance > max(10, importance of .rnorm) & is not highest
        #       combine factor dummy features to just the factor feature
            if (length(pos_rnorm <- 
                       grep(".rnorm", row.names(bst_featsimp_df), fixed=TRUE)) > 0)
                imp_rnorm <- bst_featsimp_df[pos_rnorm, 1] else
                imp_rnorm <- NA    
            imp_cutoff <- max(10, imp_rnorm, na.rm=TRUE)
            interact_vars <- 
                tail(row.names(subset(bst_featsimp_df, 
                                      imp > imp_cutoff)), -1)
            if (length(interact_vars) > 0) {
                interact_vars <-
                    myadjustInteractionFeats(glb_feats_df, myextract_actual_feats(interact_vars))
                interact_vars <- 
                    interact_vars[!grepl(topindep_var, interact_vars, fixed=TRUE)]
            }
            ### bid0_sp only
#             interact_vars <- c(
#     "biddable", "D.ratio.sum.TfIdf.wrds.n", "D.TfIdf.sum.stem.stop.Ratio", "D.sum.TfIdf",
#     "D.TfIdf.sum.post.stop", "D.TfIdf.sum.post.stem", "D.ratio.wrds.stop.n.wrds.n", "D.chrs.uppr.n.log",
#     "D.chrs.n.log", "color.fctr"
#     # , "condition.fctr", "prdl.my.descr.fctr"
#                                 )
#            interact_vars <- setdiff(interact_vars, c("startprice.dgt2.is9", "color.fctr"))
            ###
            indepVar <- myextract_actual_feats(row.names(bst_featsimp_df))
            indepVar <- setdiff(indepVar, topindep_var)
            if (length(interact_vars) > 0) {
                indepVar <- 
                    setdiff(indepVar, myextract_actual_feats(interact_vars))
                indepVar <- c(indepVar, 
                    paste(topindep_var, setdiff(interact_vars, topindep_var), 
                          sep = "*"))
            } else indepVar <- union(indepVar, topindep_var)
        }
    }
    
    if (is.null(indepVar))
        indepVar <- glb_mdl_feats_lst[[mdl_id_pfx]]

    if (is.null(indepVar) && grepl("RFE\\.", mdl_id_pfx))
        indepVar <- myextract_actual_feats(predictors(rfe_fit_results))
    
    if (is.null(indepVar))
        indepVar <- mygetIndepVar(glb_feats_df)
    
    if ((length(indepVar) == 1) && (grepl("^%<d-%", indepVar))) {    
        indepVar <- 
            eval(parse(text = str_trim(unlist(strsplit(indepVar, "%<d-%"))[2])))
    }    

    indepVar <- myadjustInteractionFeats(glb_feats_df, indepVar)
    
    if (grepl("\\.Interact", mdl_id_pfx)) { 
        # if (method != tail(unlist(strsplit(bst_mdl_id, "[.]")), 1)) next
        if (is.null(glbMdlFamilies[[mdl_id_pfx]])) {
            if (!is.null(glbMdlFamilies[["Best.Interact"]]))
                glbMdlFamilies[[mdl_id_pfx]] <-
                    glbMdlFamilies[["Best.Interact"]]
        }
    }
    
    if (!is.null(glbObsFitOutliers[[mdl_id_pfx]])) {
        fitobs_df <- glbObsFit[!(glbObsFit[, glbFeatsId] %in%
                                         glbObsFitOutliers[[mdl_id_pfx]]), ]
        print(sprintf("Outliers removed: %d", nrow(glbObsFit) - nrow(fitobs_df)))
        print(setdiff(glbObsFit[, glbFeatsId], fitobs_df[, glbFeatsId]))
    } else fitobs_df <- glbObsFit

    if (is.null(glbMdlFamilies[[mdl_id_pfx]]))
        mdl_methods <- glbMdlMethods else
        mdl_methods <- glbMdlFamilies[[mdl_id_pfx]]    

    for (method in mdl_methods) {
        if (method %in% c("rpart", "rf")) {
            # rpart:    fubar's the tree
            # rf:       skip the scenario w/ .rnorm for speed
            indepVar <- setdiff(indepVar, c(".rnorm"))
            #mdl_id <- paste0(mdl_id_pfx, ".no.rnorm")
        } 

        fit.models_1_chunk_df <- myadd_chunk(fit.models_1_chunk_df, 
                            paste0("fit.models_1_", mdl_id_pfx), major.inc = FALSE,
                                    label.minor = method)

        ret_lst <- 
            myfit_mdl(mdl_specs_lst = myinit_mdl_specs_lst(mdl_specs_lst = list(
            id.prefix = mdl_id_pfx, 
            type = glb_model_type, 
            tune.df = glbMdlTuneParams,
            trainControl.method = "repeatedcv", # or "none" if nominalWorkflow is crashing
            trainControl.number = glb_rcv_n_folds,
            trainControl.repeats = glb_rcv_n_repeats,
            trainControl.classProbs = glb_is_classification,
            trainControl.summaryFunction = glbMdlMetricSummaryFn,
            trainControl.allowParallel = glbMdlAllowParallel,            
            train.metric = glbMdlMetricSummary, 
            train.maximize = glbMdlMetricMaximize,    
            train.method = method)),
            indepVar = indepVar, rsp_var = glb_rsp_var, 
            fit_df = fitobs_df, OOB_df = glbObsOOB)
        
#         ntv_mdl <- glmnet(x = as.matrix(
#                               fitobs_df[, indepVar]), 
#                           y = as.factor(as.character(
#                               fitobs_df[, glb_rsp_var])),
#                           family = "multinomial")
#         bgn = 1; end = 100;
#         ntv_mdl <- glmnet(x = as.matrix(
#                               subset(fitobs_df, pop.fctr != "crypto")[bgn:end, indepVar]), 
#                           y = as.factor(as.character(
#                               subset(fitobs_df, pop.fctr != "crypto")[bgn:end, glb_rsp_var])),
#                           family = "multinomial")
    }
}

##                label step_major step_minor label_minor     bgn     end
## 1   fit.models_1_bgn          1          0       setup 345.908 345.919
## 2 fit.models_1_All.X          1          1       setup 345.919      NA
##   elapsed
## 1   0.011
## 2      NA
##                label step_major step_minor label_minor     bgn     end
## 2 fit.models_1_All.X          1          1       setup 345.919 345.927
## 3 fit.models_1_All.X          1          2      glmnet 345.927      NA
##   elapsed
## 2   0.008
## 3      NA
## [1] "myfit_mdl: enter: 0.000000 secs"
## [1] "fitting model: All.X##rcv#glmnet"
## [1] "    indepVar: Q109244.fctr,Hhold.fctr,Edn.fctr,Q101163.fctr,Q100689.fctr,Q98078.fctr,Q99716.fctr,Q120379.fctr,Q121699.fctr,Q105840.fctr,Q113583.fctr,Q115195.fctr,Q102089.fctr,Q98059.fctr,Q114386.fctr,Q100680.fctr,Q108342.fctr,Q111848.fctr,YOB.Age.fctr,Q118892.fctr,Q102687.fctr,Q115390.fctr,Q119851.fctr,Q114517.fctr,Q120012.fctr,Q109367.fctr,Q114961.fctr,Q121700.fctr,Q124122.fctr,Q111220.fctr,Q113992.fctr,Q121011.fctr,Q106042.fctr,Q116448.fctr,Q116601.fctr,Q104996.fctr,Q102906.fctr,Q113584.fctr,Q108950.fctr,Q102674.fctr,Q103293.fctr,Q112478.fctr,Q114748.fctr,Q107491.fctr,Q100562.fctr,Q108617.fctr,Q100010.fctr,Q115602.fctr,Q116953.fctr,Q115610.fctr,Q106997.fctr,Q120978.fctr,Q112512.fctr,Q108343.fctr,Q96024.fctr,Q106389.fctr,.rnorm,Q108754.fctr,Q98578.fctr,Q101162.fctr,Q115777.fctr,Q99581.fctr,Q124742.fctr,Q116797.fctr,Q112270.fctr,Q118237.fctr,Q119650.fctr,Q111580.fctr,Q123464.fctr,Q117193.fctr,Q99982.fctr,Q108856.fctr,Q118233.fctr,Q102289.fctr,Q116197.fctr,Income.fctr,Q118232.fctr,Q120194.fctr,Q114152.fctr,Q122770.fctr,Q117186.fctr,Q105655.fctr,Q106993.fctr,Q119334.fctr,Q122120.fctr,Q116441.fctr,Q118117.fctr,Q123621.fctr,Q122769.fctr,Q120650.fctr,Q98869.fctr,Q107869.fctr,Q120014.fctr,Q115899.fctr,Q106388.fctr,Q99480.fctr,Q122771.fctr,Q108855.fctr,Q110740.fctr,Q106272.fctr,Q101596.fctr,Q116881.fctr,Q120472.fctr,Q98197.fctr,Q113181.fctr,Q115611.fctr,Gender.fctr,Hhold.fctr:.clusterid.fctr"
## [1] "myfit_mdl: setup complete: 0.695000 secs"
## Aggregating results
## Selecting tuning parameters
## Fitting alpha = 0.325, lambda = 0.0534 on full training set
## [1] "myfit_mdl: train complete: 24.001000 secs"

## Warning in myfit_mdl(mdl_specs_lst = myinit_mdl_specs_lst(mdl_specs_lst
## = list(id.prefix = mdl_id_pfx, : model's bestTune found at an extreme of
## tuneGrid for parameter: lambda

##             Length Class      Mode     
## a0             78  -none-     numeric  
## beta        19734  dgCMatrix  S4       
## df             78  -none-     numeric  
## dim             2  -none-     numeric  
## lambda         78  -none-     numeric  
## dev.ratio      78  -none-     numeric  
## nulldev         1  -none-     numeric  
## npasses         1  -none-     numeric  
## jerr            1  -none-     numeric  
## offset          1  -none-     logical  
## classnames      2  -none-     character
## call            5  -none-     call     
## nobs            1  -none-     numeric  
## lambdaOpt       1  -none-     numeric  
## xNames        253  -none-     character
## problemType     1  -none-     character
## tuneValue       2  data.frame list     
## obsLevels       2  -none-     character
## [1] "min lambda > lambdaOpt:"
##       (Intercept)      Gender.fctrM     Hhold.fctrMKy     Hhold.fctrPKn 
##      0.1759341202     -0.0577396529     -0.0655451865      0.3282606962 
##   Q101163.fctrDad   Q101163.fctrMom    Q109244.fctrNo   Q109244.fctrYes 
##     -0.0462347606      0.0361283260     -0.3170070971      0.7976156342 
##    Q110740.fctrPC    Q113181.fctrNo   Q113181.fctrYes    Q115611.fctrNo 
##     -0.0120813108      0.0348921948     -0.0798654082      0.0770473552 
##   Q115611.fctrYes Q116881.fctrHappy Q116881.fctrRight    Q118232.fctrId 
##     -0.3105852699      0.0087382029     -0.0814535814      0.0180660706 
##    Q119851.fctrNo    Q120379.fctrNo   Q120379.fctrYes     Q98197.fctrNo 
##     -0.0297897358     -0.0006993182      0.0272478036      0.1639383807 
##    Q98197.fctrYes     Q98869.fctrNo     Q99480.fctrNo 
##     -0.0341915807      0.1609634790      0.0576623212 
## [1] "max lambda < lambdaOpt:"
##       (Intercept)      Gender.fctrM     Hhold.fctrMKy     Hhold.fctrPKn 
##       0.180590790      -0.063356510      -0.071963008       0.358397790 
##     Income.fctr.Q   Q101163.fctrDad   Q101163.fctrMom    Q109244.fctrNo 
##      -0.003003778      -0.055383160       0.041352778      -0.326547933 
##   Q109244.fctrYes    Q110740.fctrPC    Q113181.fctrNo   Q113181.fctrYes 
##       0.811987955      -0.023695743       0.041178168      -0.082431818 
##    Q115611.fctrNo   Q115611.fctrYes    Q115899.fctrCs Q116881.fctrHappy 
##       0.080608650      -0.318215302       0.001114558       0.019171363 
## Q116881.fctrRight    Q118232.fctrId    Q119851.fctrNo    Q120379.fctrNo 
##      -0.090653797       0.032794176      -0.043521175      -0.004314002 
##   Q120379.fctrYes    Q121699.fctrNo    Q122771.fctrPt     Q98197.fctrNo 
##       0.038379682      -0.010439127      -0.000907088       0.171596860 
##    Q98197.fctrYes     Q98869.fctrNo     Q99480.fctrNo 
##      -0.034226771       0.174722108       0.068918234 
## [1] "myfit_mdl: train diagnostics complete: 24.795000 secs"

##          Prediction
## Reference    R    D
##         R 1911  180
##         D 1655  702
##       Accuracy          Kappa  AccuracyLower  AccuracyUpper   AccuracyNull 
##   5.874550e-01   2.036470e-01   5.728218e-01   6.019734e-01   5.299011e-01 
## AccuracyPValue  McnemarPValue 
##   6.718265e-15  1.812597e-259

##          Prediction
## Reference   R   D
##         R 494  32
##         D 467 127
##       Accuracy          Kappa  AccuracyLower  AccuracyUpper   AccuracyNull 
##   5.544643e-01   1.460545e-01   5.247985e-01   5.838432e-01   5.303571e-01 
## AccuracyPValue  McnemarPValue 
##   5.617624e-02   4.430516e-84 
## [1] "myfit_mdl: predict complete: 34.976000 secs"
##                  id
## 1 All.X##rcv#glmnet
##                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               feats
## 1 Q109244.fctr,Hhold.fctr,Edn.fctr,Q101163.fctr,Q100689.fctr,Q98078.fctr,Q99716.fctr,Q120379.fctr,Q121699.fctr,Q105840.fctr,Q113583.fctr,Q115195.fctr,Q102089.fctr,Q98059.fctr,Q114386.fctr,Q100680.fctr,Q108342.fctr,Q111848.fctr,YOB.Age.fctr,Q118892.fctr,Q102687.fctr,Q115390.fctr,Q119851.fctr,Q114517.fctr,Q120012.fctr,Q109367.fctr,Q114961.fctr,Q121700.fctr,Q124122.fctr,Q111220.fctr,Q113992.fctr,Q121011.fctr,Q106042.fctr,Q116448.fctr,Q116601.fctr,Q104996.fctr,Q102906.fctr,Q113584.fctr,Q108950.fctr,Q102674.fctr,Q103293.fctr,Q112478.fctr,Q114748.fctr,Q107491.fctr,Q100562.fctr,Q108617.fctr,Q100010.fctr,Q115602.fctr,Q116953.fctr,Q115610.fctr,Q106997.fctr,Q120978.fctr,Q112512.fctr,Q108343.fctr,Q96024.fctr,Q106389.fctr,.rnorm,Q108754.fctr,Q98578.fctr,Q101162.fctr,Q115777.fctr,Q99581.fctr,Q124742.fctr,Q116797.fctr,Q112270.fctr,Q118237.fctr,Q119650.fctr,Q111580.fctr,Q123464.fctr,Q117193.fctr,Q99982.fctr,Q108856.fctr,Q118233.fctr,Q102289.fctr,Q116197.fctr,Income.fctr,Q118232.fctr,Q120194.fctr,Q114152.fctr,Q122770.fctr,Q117186.fctr,Q105655.fctr,Q106993.fctr,Q119334.fctr,Q122120.fctr,Q116441.fctr,Q118117.fctr,Q123621.fctr,Q122769.fctr,Q120650.fctr,Q98869.fctr,Q107869.fctr,Q120014.fctr,Q115899.fctr,Q106388.fctr,Q99480.fctr,Q122771.fctr,Q108855.fctr,Q110740.fctr,Q106272.fctr,Q101596.fctr,Q116881.fctr,Q120472.fctr,Q98197.fctr,Q113181.fctr,Q115611.fctr,Gender.fctr,Hhold.fctr:.clusterid.fctr
##   max.nTuningRuns min.elapsedtime.everything min.elapsedtime.final
## 1              25                     23.189                 2.025
##   max.AUCpROC.fit max.Sens.fit max.Spec.fit max.AUCROCR.fit
## 1       0.6273089    0.5155428    0.7390751       0.3048695
##   opt.prob.threshold.fit max.f.score.fit max.Accuracy.fit
## 1                    0.6       0.6756231        0.6241754
##   max.AccuracyLower.fit max.AccuracyUpper.fit max.Kappa.fit
## 1             0.5728218             0.6019734     0.2388455
##   max.AUCpROC.OOB max.Sens.OOB max.Spec.OOB max.AUCROCR.OOB
## 1       0.6237374          0.5    0.7474747       0.3184875
##   opt.prob.threshold.OOB max.f.score.OOB max.Accuracy.OOB
## 1                    0.7        0.664425        0.5544643
##   max.AccuracyLower.OOB max.AccuracyUpper.OOB max.Kappa.OOB
## 1             0.5247985             0.5838432     0.1460545
##   max.AccuracySD.fit max.KappaSD.fit
## 1        0.004938833      0.01189291
## [1] "myfit_mdl: exit: 34.992000 secs"
##                label step_major step_minor label_minor     bgn     end
## 3 fit.models_1_All.X          1          2      glmnet 345.927 380.924
## 4 fit.models_1_All.X          1          3         glm 380.925      NA
##   elapsed
## 3  34.997
## 4      NA
## [1] "myfit_mdl: enter: 0.000000 secs"
## [1] "fitting model: All.X##rcv#glm"
## [1] "    indepVar: Q109244.fctr,Hhold.fctr,Edn.fctr,Q101163.fctr,Q100689.fctr,Q98078.fctr,Q99716.fctr,Q120379.fctr,Q121699.fctr,Q105840.fctr,Q113583.fctr,Q115195.fctr,Q102089.fctr,Q98059.fctr,Q114386.fctr,Q100680.fctr,Q108342.fctr,Q111848.fctr,YOB.Age.fctr,Q118892.fctr,Q102687.fctr,Q115390.fctr,Q119851.fctr,Q114517.fctr,Q120012.fctr,Q109367.fctr,Q114961.fctr,Q121700.fctr,Q124122.fctr,Q111220.fctr,Q113992.fctr,Q121011.fctr,Q106042.fctr,Q116448.fctr,Q116601.fctr,Q104996.fctr,Q102906.fctr,Q113584.fctr,Q108950.fctr,Q102674.fctr,Q103293.fctr,Q112478.fctr,Q114748.fctr,Q107491.fctr,Q100562.fctr,Q108617.fctr,Q100010.fctr,Q115602.fctr,Q116953.fctr,Q115610.fctr,Q106997.fctr,Q120978.fctr,Q112512.fctr,Q108343.fctr,Q96024.fctr,Q106389.fctr,.rnorm,Q108754.fctr,Q98578.fctr,Q101162.fctr,Q115777.fctr,Q99581.fctr,Q124742.fctr,Q116797.fctr,Q112270.fctr,Q118237.fctr,Q119650.fctr,Q111580.fctr,Q123464.fctr,Q117193.fctr,Q99982.fctr,Q108856.fctr,Q118233.fctr,Q102289.fctr,Q116197.fctr,Income.fctr,Q118232.fctr,Q120194.fctr,Q114152.fctr,Q122770.fctr,Q117186.fctr,Q105655.fctr,Q106993.fctr,Q119334.fctr,Q122120.fctr,Q116441.fctr,Q118117.fctr,Q123621.fctr,Q122769.fctr,Q120650.fctr,Q98869.fctr,Q107869.fctr,Q120014.fctr,Q115899.fctr,Q106388.fctr,Q99480.fctr,Q122771.fctr,Q108855.fctr,Q110740.fctr,Q106272.fctr,Q101596.fctr,Q116881.fctr,Q120472.fctr,Q98197.fctr,Q113181.fctr,Q115611.fctr,Gender.fctr,Hhold.fctr:.clusterid.fctr"
## [1] "myfit_mdl: setup complete: 0.719000 secs"
## + Fold1.Rep1: parameter=none

## Warning in predict.lm(object, newdata, se.fit, scale = 1, type =
## ifelse(type == : prediction from a rank-deficient fit may be misleading

## - Fold1.Rep1: parameter=none 
## + Fold2.Rep1: parameter=none

## Warning in predict.lm(object, newdata, se.fit, scale = 1, type =
## ifelse(type == : prediction from a rank-deficient fit may be misleading

## - Fold2.Rep1: parameter=none 
## + Fold3.Rep1: parameter=none

## Warning in predict.lm(object, newdata, se.fit, scale = 1, type =
## ifelse(type == : prediction from a rank-deficient fit may be misleading

## - Fold3.Rep1: parameter=none 
## + Fold1.Rep2: parameter=none

## Warning in predict.lm(object, newdata, se.fit, scale = 1, type =
## ifelse(type == : prediction from a rank-deficient fit may be misleading

## - Fold1.Rep2: parameter=none 
## + Fold2.Rep2: parameter=none

## Warning in predict.lm(object, newdata, se.fit, scale = 1, type =
## ifelse(type == : prediction from a rank-deficient fit may be misleading

## - Fold2.Rep2: parameter=none 
## + Fold3.Rep2: parameter=none

## Warning in predict.lm(object, newdata, se.fit, scale = 1, type =
## ifelse(type == : prediction from a rank-deficient fit may be misleading

## - Fold3.Rep2: parameter=none 
## + Fold1.Rep3: parameter=none

## Warning in predict.lm(object, newdata, se.fit, scale = 1, type =
## ifelse(type == : prediction from a rank-deficient fit may be misleading

## - Fold1.Rep3: parameter=none 
## + Fold2.Rep3: parameter=none

## Warning in predict.lm(object, newdata, se.fit, scale = 1, type =
## ifelse(type == : prediction from a rank-deficient fit may be misleading

## - Fold2.Rep3: parameter=none 
## + Fold3.Rep3: parameter=none

## Warning in predict.lm(object, newdata, se.fit, scale = 1, type =
## ifelse(type == : prediction from a rank-deficient fit may be misleading

## - Fold3.Rep3: parameter=none 
## Aggregating results
## Fitting final model on full training set
## [1] "myfit_mdl: train complete: 20.541000 secs"

## 
## Call:
## NULL
## 
## Deviance Residuals: 
##     Min       1Q   Median       3Q      Max  
## -3.0785  -1.0322   0.4086   1.0348   2.3700  
## 
## Coefficients: (5 not defined because of singularities)
##                                    Estimate Std. Error z value Pr(>|z|)
## (Intercept)                       0.4863587  0.3111062   1.563 0.117977
## .rnorm                           -0.0184823  0.0335919  -0.550 0.582181
## Edn.fctr.L                       -0.0635253  0.1663643  -0.382 0.702577
## Edn.fctr.Q                       -0.0041131  0.1465736  -0.028 0.977613
## Edn.fctr.C                       -0.0025043  0.1280728  -0.020 0.984400
## `Edn.fctr^4`                     -0.3512568  0.1264805  -2.777 0.005484
## `Edn.fctr^5`                     -0.0637218  0.1168470  -0.545 0.585517
## `Edn.fctr^6`                      0.1507290  0.1058000   1.425 0.154255
## `Edn.fctr^7`                      0.1916553  0.1171629   1.636 0.101881
## Gender.fctrF                     -0.4141680  0.2451966  -1.689 0.091195
## Gender.fctrM                     -0.4838132  0.2421867  -1.998 0.045751
## Hhold.fctrMKn                    -0.0226144  0.2488967  -0.091 0.927605
## Hhold.fctrMKy                    -0.1780372  0.2274646  -0.783 0.433802
## Hhold.fctrPKn                     1.0282794  0.3358894   3.061 0.002203
## Hhold.fctrPKy                     0.1290149  0.5993871   0.215 0.829577
## Hhold.fctrSKn                     0.1116977  0.2168416   0.515 0.606475
## Hhold.fctrSKy                     0.6764977  0.3423382   1.976 0.048142
## Income.fctr.L                    -0.0964692  0.1081466  -0.892 0.372381
## Income.fctr.Q                    -0.1798166  0.0994266  -1.809 0.070523
## Income.fctr.C                    -0.2586068  0.0967468  -2.673 0.007517
## `Income.fctr^4`                  -0.1092232  0.0939673  -1.162 0.245092
## `Income.fctr^5`                  -0.0119522  0.0956573  -0.125 0.900564
## `Income.fctr^6`                   0.0919866  0.0932579   0.986 0.323953
## Q100010.fctrNo                    0.2638160  0.2176291   1.212 0.225425
## Q100010.fctrYes                   0.1835542  0.1993828   0.921 0.357253
## Q100562.fctrNo                    0.0486364  0.2008690   0.242 0.808679
## Q100562.fctrYes                   0.0557525  0.1767182   0.315 0.752391
## Q100680.fctrNo                   -0.2097049  0.1932815  -1.085 0.277934
## Q100680.fctrYes                  -0.1561799  0.1869812  -0.835 0.403565
## Q100689.fctrNo                    0.4052609  0.1950878   2.077 0.037772
## Q100689.fctrYes                   0.5573111  0.1935986   2.879 0.003993
## Q101162.fctrOptimist              0.0682476  0.1799966   0.379 0.704569
## Q101162.fctrPessimist             0.0502801  0.1861276   0.270 0.787054
## Q101163.fctrDad                  -0.2436710  0.1593750  -1.529 0.126285
## Q101163.fctrMom                   0.0395320  0.1636616   0.242 0.809131
## Q101596.fctrNo                   -0.4847887  0.1637602  -2.960 0.003073
## Q101596.fctrYes                  -0.4482783  0.1732100  -2.588 0.009652
## Q102089.fctrOwn                   0.1442032  0.1648768   0.875 0.381785
## Q102089.fctrRent                  0.1160827  0.1741246   0.667 0.504986
## Q102289.fctrNo                    0.0928191  0.1686551   0.550 0.582080
## Q102289.fctrYes                   0.0396850  0.1794951   0.221 0.825021
## Q102674.fctrNo                   -0.4074912  0.2170933  -1.877 0.060514
## Q102674.fctrYes                  -0.3693772  0.2283356  -1.618 0.105728
## Q102687.fctrNo                    0.4089850  0.2315929   1.766 0.077402
## Q102687.fctrYes                   0.4661745  0.2297042   2.029 0.042412
## Q102906.fctrNo                    0.0148766  0.1709761   0.087 0.930664
## Q102906.fctrYes                  -0.0191922  0.1755198  -0.109 0.912929
## Q103293.fctrNo                   -0.0987477  0.1534346  -0.644 0.519847
## Q103293.fctrYes                   0.0029749  0.1551789   0.019 0.984705
## Q104996.fctrNo                   -0.0210770  0.1435951  -0.147 0.883305
## Q104996.fctrYes                   0.1373682  0.1419276   0.968 0.333107
## Q105655.fctrNo                   -0.0870665  0.1746500  -0.499 0.618118
## Q105655.fctrYes                  -0.1757525  0.1725515  -1.019 0.308416
## Q105840.fctrNo                    0.0696870  0.1758568   0.396 0.691905
## Q105840.fctrYes                   0.0604114  0.1770139   0.341 0.732892
## Q106042.fctrNo                   -0.2348935  0.1745573  -1.346 0.178415
## Q106042.fctrYes                  -0.1811104  0.1748850  -1.036 0.300390
## Q106272.fctrNo                    0.1756798  0.1956044   0.898 0.369112
## Q106272.fctrYes                   0.0227920  0.1820743   0.125 0.900381
## Q106388.fctrNo                   -0.0331275  0.2138446  -0.155 0.876889
## Q106388.fctrYes                  -0.0168889  0.2261490  -0.075 0.940469
## Q106389.fctrNo                   -0.2505615  0.2123127  -1.180 0.237939
## Q106389.fctrYes                  -0.0764743  0.2140269  -0.357 0.720859
## Q106993.fctrNo                   -0.2335449  0.2156682  -1.083 0.278857
## Q106993.fctrYes                  -0.1130429  0.1918245  -0.589 0.555657
## Q106997.fctrGr                    0.0149705  0.1936824   0.077 0.938390
## Q106997.fctrYy                    0.2671127  0.1972202   1.354 0.175613
## Q107491.fctrNo                    0.0799487  0.1821461   0.439 0.660715
## Q107491.fctrYes                   0.1528622  0.1392683   1.098 0.272375
## Q107869.fctrNo                    0.0160630  0.1464693   0.110 0.912673
## Q107869.fctrYes                  -0.0602896  0.1471670  -0.410 0.682050
## `Q108342.fctrIn-person`           0.2378940  0.1755809   1.355 0.175450
## Q108342.fctrOnline                0.3833999  0.1854190   2.068 0.038664
## Q108343.fctrNo                   -0.0837014  0.1820387  -0.460 0.645660
## Q108343.fctrYes                  -0.1527770  0.1922880  -0.795 0.426892
## Q108617.fctrNo                    0.0781605  0.1665499   0.469 0.638861
## Q108617.fctrYes                  -0.1349061  0.2091038  -0.645 0.518821
## Q108754.fctrNo                    0.0844578  0.1874623   0.451 0.652327
## Q108754.fctrYes                   0.0825214  0.1961834   0.421 0.674022
## Q108855.fctrUmm...               -0.0639760  0.2112984  -0.303 0.762061
## `Q108855.fctrYes!`               -0.1888330  0.2076284  -0.909 0.363099
## Q108856.fctrSocialize            -0.1921966  0.2140900  -0.898 0.369326
## Q108856.fctrSpace                -0.2055906  0.1996494  -1.030 0.303124
## Q108950.fctrCautious              0.1052653  0.1565545   0.672 0.501337
## `Q108950.fctrRisk-friendly`       0.2177849  0.1678652   1.297 0.194501
## Q109244.fctrNo                   -0.5664914  0.1558879  -3.634 0.000279
## Q109244.fctrYes                   0.9209911  0.2002998   4.598 4.26e-06
## Q109367.fctrNo                    0.1329450  0.1540848   0.863 0.388245
## Q109367.fctrYes                   0.0795031  0.1474211   0.539 0.589685
## Q110740.fctrMac                  -0.0106319  0.1303334  -0.082 0.934985
## Q110740.fctrPC                   -0.2181024  0.1274423  -1.711 0.087011
## Q111220.fctrNo                   -0.0188704  0.1401573  -0.135 0.892899
## Q111220.fctrYes                   0.1867903  0.1536053   1.216 0.223969
## Q111580.fctrDemanding            -0.0303561  0.1537804  -0.197 0.843515
## Q111580.fctrSupportive            0.0036072  0.1441579   0.025 0.980037
## Q111848.fctrNo                    0.1263334  0.1524437   0.829 0.407262
## Q111848.fctrYes                   0.1472391  0.1473187   0.999 0.317572
## Q112270.fctrNo                    0.1354788  0.1427256   0.949 0.342506
## Q112270.fctrYes                   0.1805332  0.1428220   1.264 0.206215
## Q112478.fctrNo                   -0.3489845  0.1742727  -2.003 0.045229
## Q112478.fctrYes                  -0.1473014  0.1681200  -0.876 0.380939
## Q112512.fctrNo                    0.0710991  0.1845700   0.385 0.700078
## Q112512.fctrYes                   0.0146547  0.1577724   0.093 0.925995
## Q113181.fctrNo                    0.0847720  0.1384544   0.612 0.540357
## Q113181.fctrYes                  -0.2044321  0.1448979  -1.411 0.158283
## Q113583.fctrTalk                  0.0840476  0.1979759   0.425 0.671176
## Q113583.fctrTunes                 0.1204067  0.1901213   0.633 0.526528
## Q113584.fctrPeople               -0.1249798  0.1947365  -0.642 0.521010
## Q113584.fctrTechnology           -0.1087634  0.1936061  -0.562 0.574268
## Q113992.fctrNo                    0.1933146  0.1556541   1.242 0.214255
## Q113992.fctrYes                   0.2805266  0.1667975   1.682 0.092600
## Q114152.fctrNo                   -0.1268258  0.1526474  -0.831 0.406063
## Q114152.fctrYes                  -0.0075338  0.1639645  -0.046 0.963352
## Q114386.fctrMysterious            0.0493940  0.1543878   0.320 0.749018
## Q114386.fctrTMI                  -0.0277319  0.1577795  -0.176 0.860480
## Q114517.fctrNo                    0.1918425  0.1674246   1.146 0.251860
## Q114517.fctrYes                   0.2125597  0.1778709   1.195 0.232078
## Q114748.fctrNo                   -0.3216896  0.1774907  -1.812 0.069920
## Q114748.fctrYes                  -0.2884521  0.1756052  -1.643 0.100462
## Q114961.fctrNo                    0.2065555  0.1698519   1.216 0.223950
## Q114961.fctrYes                   0.1575295  0.1687244   0.934 0.350484
## Q115195.fctrNo                    0.0797354  0.1671389   0.477 0.633319
## Q115195.fctrYes                   0.1086965  0.1571798   0.692 0.489225
## Q115390.fctrNo                   -0.2090270  0.1505530  -1.388 0.165017
## Q115390.fctrYes                   0.0010154  0.1411008   0.007 0.994258
## Q115602.fctrNo                    0.0752417  0.1938875   0.388 0.697965
## Q115602.fctrYes                   0.1885997  0.1733086   1.088 0.276493
## Q115610.fctrNo                   -0.0540633  0.2046213  -0.264 0.791617
## Q115610.fctrYes                  -0.0525227  0.1810262  -0.290 0.771710
## Q115611.fctrNo                   -0.0434473  0.1915160  -0.227 0.820533
## Q115611.fctrYes                  -0.6105058  0.1974971  -3.091 0.001993
## Q115777.fctrEnd                   0.0012892  0.1607489   0.008 0.993601
## Q115777.fctrStart                 0.0611859  0.1567451   0.390 0.696276
## Q115899.fctrCs                    0.2138916  0.1580743   1.353 0.176021
## Q115899.fctrMe                    0.0186769  0.1559377   0.120 0.904664
## Q116197.fctrA.M.                 -0.3812902  0.1568391  -2.431 0.015053
## Q116197.fctrP.M.                 -0.2696720  0.1462783  -1.844 0.065248
## Q116441.fctrNo                   -0.1815864  0.1770582  -1.026 0.305092
## Q116441.fctrYes                  -0.1064025  0.1904685  -0.559 0.576410
## Q116448.fctrNo                    0.1849214  0.1679901   1.101 0.270989
## Q116448.fctrYes                   0.1412910  0.1691868   0.835 0.403651
## Q116601.fctrNo                    0.2227314  0.1968951   1.131 0.257963
## Q116601.fctrYes                   0.1813233  0.1681914   1.078 0.280999
## Q116797.fctrNo                   -0.1507811  0.1698059  -0.888 0.374561
## Q116797.fctrYes                  -0.1901829  0.1750157  -1.087 0.277186
## Q116881.fctrHappy                 0.1409183  0.1654665   0.852 0.394413
## Q116881.fctrRight                -0.1920428  0.1804795  -1.064 0.287297
## Q116953.fctrNo                    0.0145321  0.1773701   0.082 0.934701
## Q116953.fctrYes                   0.2566470  0.1668224   1.538 0.123940
## `Q117186.fctrCool headed`         0.0072858  0.1650996   0.044 0.964801
## `Q117186.fctrHot headed`         -0.0872428  0.1733050  -0.503 0.614679
## `Q117193.fctrOdd hours`           0.0007539  0.1621574   0.005 0.996291
## `Q117193.fctrStandard hours`     -0.0833027  0.1545419  -0.539 0.589866
## Q118117.fctrNo                   -0.0304918  0.1494757  -0.204 0.838360
## Q118117.fctrYes                  -0.0050857  0.1516619  -0.034 0.973249
## Q118232.fctrId                    0.4155562  0.1478075   2.811 0.004932
## Q118232.fctrPr                    0.2226338  0.1459558   1.525 0.127172
## Q118233.fctrNo                   -0.1483453  0.1872100  -0.792 0.428127
## Q118233.fctrYes                   0.0196958  0.2032659   0.097 0.922808
## Q118237.fctrNo                   -0.1381557  0.1898795  -0.728 0.466861
## Q118237.fctrYes                  -0.1142702  0.1867887  -0.612 0.540695
## Q118892.fctrNo                    0.0958753  0.1327467   0.722 0.470145
## Q118892.fctrYes                   0.0687583  0.1253402   0.549 0.583299
## Q119334.fctrNo                   -0.1315869  0.1371446  -0.959 0.337319
## Q119334.fctrYes                  -0.1089266  0.1336240  -0.815 0.414974
## Q119650.fctrGiving               -0.1144200  0.1419043  -0.806 0.420059
## Q119650.fctrReceiving            -0.0001615  0.1586196  -0.001 0.999188
## Q119851.fctrNo                   -0.1871074  0.1634382  -1.145 0.252284
## Q119851.fctrYes                  -0.0276015  0.1625929  -0.170 0.865200
## Q120012.fctrNo                    0.0521281  0.1625359   0.321 0.748425
## Q120012.fctrYes                   0.1526382  0.1613759   0.946 0.344223
## Q120014.fctrNo                    0.0147040  0.1513326   0.097 0.922596
## Q120014.fctrYes                  -0.1183100  0.1437549  -0.823 0.410509
## `Q120194.fctrStudy first`         0.3151826  0.1391360   2.265 0.023495
## `Q120194.fctrTry first`           0.2179022  0.1444844   1.508 0.131520
## Q120379.fctrNo                   -0.0717323  0.1527075  -0.470 0.638543
## Q120379.fctrYes                   0.2271637  0.1510707   1.504 0.132661
## Q120472.fctrArt                  -0.0658720  0.1553248  -0.424 0.671499
## Q120472.fctrScience              -0.1444360  0.1451121  -0.995 0.319570
## Q120650.fctrNo                   -0.0583824  0.1974030  -0.296 0.767419
## Q120650.fctrYes                  -0.2069377  0.1444083  -1.433 0.151857
## Q120978.fctrNo                    0.0653400  0.1585173   0.412 0.680197
## Q120978.fctrYes                   0.0681184  0.1547709   0.440 0.659847
## Q121011.fctrNo                    0.1749610  0.1591772   1.099 0.271699
## Q121011.fctrYes                   0.1466920  0.1566037   0.937 0.348908
## Q121699.fctrNo                    0.3914766  0.2438850   1.605 0.108457
## Q121699.fctrYes                   0.5682661  0.2350022   2.418 0.015600
## Q121700.fctrNo                   -0.4414275  0.2373708  -1.860 0.062935
## Q121700.fctrYes                  -0.3533231  0.2559683  -1.380 0.167482
## Q122120.fctrNo                   -0.0472010  0.1388951  -0.340 0.733983
## Q122120.fctrYes                  -0.1302761  0.1526728  -0.853 0.393492
## Q122769.fctrNo                   -0.0733511  0.2115704  -0.347 0.728818
## Q122769.fctrYes                  -0.0616149  0.2144801  -0.287 0.773901
## Q122770.fctrNo                    0.1714970  0.2564315   0.669 0.503634
## Q122770.fctrYes                   0.1426217  0.2530003   0.564 0.572944
## Q122771.fctrPc                   -0.2132080  0.2348521  -0.908 0.363963
## Q122771.fctrPt                   -0.4267219  0.2490691  -1.713 0.086663
## Q123464.fctrNo                   -0.0839788  0.1599206  -0.525 0.599494
## Q123464.fctrYes                   0.0689629  0.2345722   0.294 0.768762
## Q123621.fctrNo                   -0.0457153  0.1654487  -0.276 0.782309
## Q123621.fctrYes                  -0.0176390  0.1705014  -0.103 0.917603
## Q124122.fctrNo                   -0.0442878  0.1365860  -0.324 0.745750
## Q124122.fctrYes                   0.1059992  0.1311534   0.808 0.418971
## Q124742.fctrNo                    0.1714825  0.1049041   1.635 0.102120
## Q124742.fctrYes                   0.0105159  0.1212525   0.087 0.930888
## Q96024.fctrNo                     0.0810819  0.1335707   0.607 0.543829
## Q96024.fctrYes                    0.0242529  0.1246323   0.195 0.845710
## `Q98059.fctrOnly-child`           0.0107149  0.2490538   0.043 0.965684
## Q98059.fctrYes                    0.2992686  0.2082894   1.437 0.150777
## Q98078.fctrNo                    -0.0977575  0.1924671  -0.508 0.611511
## Q98078.fctrYes                   -0.1491210  0.1949939  -0.765 0.444422
## Q98197.fctrNo                     0.4033074  0.1881727   2.143 0.032090
## Q98197.fctrYes                    0.0407554  0.1936925   0.210 0.833345
## Q98578.fctrNo                    -0.4015857  0.1560438  -2.574 0.010066
## Q98578.fctrYes                   -0.2811608  0.1632185  -1.723 0.084960
## Q98869.fctrNo                     0.4898851  0.1692816   2.894 0.003805
## Q98869.fctrYes                    0.0729790  0.1436467   0.508 0.611422
## Q99480.fctrNo                     0.1398033  0.2040617   0.685 0.493279
## Q99480.fctrYes                   -0.1597246  0.1866395  -0.856 0.392113
## Q99581.fctrNo                    -0.2118357  0.2021761  -1.048 0.294741
## Q99581.fctrYes                   -0.1591203  0.2288143  -0.695 0.486797
## Q99716.fctrNo                     0.2675050  0.1741442   1.536 0.124511
## Q99716.fctrYes                    0.2091876  0.2239424   0.934 0.350245
## `Q99982.fctrCheck!`              -0.2312810  0.1940312  -1.192 0.233270
## Q99982.fctrNope                  -0.1465700  0.1969762  -0.744 0.456816
## YOB.Age.fctr.L                    0.5180159  0.1924042   2.692 0.007095
## YOB.Age.fctr.Q                    0.2487963  0.1580285   1.574 0.115400
## YOB.Age.fctr.C                   -0.0587920  0.1368797  -0.430 0.667548
## `YOB.Age.fctr^4`                  0.2548138  0.1279684   1.991 0.046456
## `YOB.Age.fctr^5`                  0.0878936  0.1181947   0.744 0.457098
## `YOB.Age.fctr^6`                  0.1426167  0.1065142   1.339 0.180589
## `YOB.Age.fctr^7`                 -0.1860942  0.1011298  -1.840 0.065746
## `YOB.Age.fctr^8`                 -0.2121412  0.1042615  -2.035 0.041881
## `Hhold.fctrN:.clusterid.fctr2`   -0.0370534  0.2729743  -0.136 0.892027
## `Hhold.fctrMKn:.clusterid.fctr2` -0.0492387  0.2528959  -0.195 0.845628
## `Hhold.fctrMKy:.clusterid.fctr2` -0.0410318  0.1935031  -0.212 0.832070
## `Hhold.fctrPKn:.clusterid.fctr2` -0.4313048  0.5305580  -0.813 0.416260
## `Hhold.fctrPKy:.clusterid.fctr2` -0.3469013  0.8011541  -0.433 0.665013
## `Hhold.fctrSKn:.clusterid.fctr2` -0.1225375  0.1565120  -0.783 0.433670
## `Hhold.fctrSKy:.clusterid.fctr2` -1.0639676  0.5030727  -2.115 0.034435
## `Hhold.fctrN:.clusterid.fctr3`   -0.2121783  0.3161575  -0.671 0.502147
## `Hhold.fctrMKn:.clusterid.fctr3` -0.0014445  0.3316501  -0.004 0.996525
## `Hhold.fctrMKy:.clusterid.fctr3` -0.0719980  0.2106200  -0.342 0.732473
## `Hhold.fctrPKn:.clusterid.fctr3` -0.4466802  0.5715377  -0.782 0.434484
## `Hhold.fctrPKy:.clusterid.fctr3` -0.2826249  0.8460983  -0.334 0.738355
## `Hhold.fctrSKn:.clusterid.fctr3`  0.0973551  0.1782935   0.546 0.585040
## `Hhold.fctrSKy:.clusterid.fctr3` -0.6723109  0.4967030  -1.354 0.175881
## `Hhold.fctrN:.clusterid.fctr4`           NA         NA      NA       NA
## `Hhold.fctrMKn:.clusterid.fctr4`         NA         NA      NA       NA
## `Hhold.fctrMKy:.clusterid.fctr4`         NA         NA      NA       NA
## `Hhold.fctrPKn:.clusterid.fctr4`         NA         NA      NA       NA
## `Hhold.fctrPKy:.clusterid.fctr4`  1.5215754  1.2184560   1.249 0.211748
## `Hhold.fctrSKn:.clusterid.fctr4`  0.1577533  0.1840584   0.857 0.391399
## `Hhold.fctrSKy:.clusterid.fctr4`         NA         NA      NA       NA
##                                     
## (Intercept)                         
## .rnorm                              
## Edn.fctr.L                          
## Edn.fctr.Q                          
## Edn.fctr.C                          
## `Edn.fctr^4`                     ** 
## `Edn.fctr^5`                        
## `Edn.fctr^6`                        
## `Edn.fctr^7`                        
## Gender.fctrF                     .  
## Gender.fctrM                     *  
## Hhold.fctrMKn                       
## Hhold.fctrMKy                       
## Hhold.fctrPKn                    ** 
## Hhold.fctrPKy                       
## Hhold.fctrSKn                       
## Hhold.fctrSKy                    *  
## Income.fctr.L                       
## Income.fctr.Q                    .  
## Income.fctr.C                    ** 
## `Income.fctr^4`                     
## `Income.fctr^5`                     
## `Income.fctr^6`                     
## Q100010.fctrNo                      
## Q100010.fctrYes                     
## Q100562.fctrNo                      
## Q100562.fctrYes                     
## Q100680.fctrNo                      
## Q100680.fctrYes                     
## Q100689.fctrNo                   *  
## Q100689.fctrYes                  ** 
## Q101162.fctrOptimist                
## Q101162.fctrPessimist               
## Q101163.fctrDad                     
## Q101163.fctrMom                     
## Q101596.fctrNo                   ** 
## Q101596.fctrYes                  ** 
## Q102089.fctrOwn                     
## Q102089.fctrRent                    
## Q102289.fctrNo                      
## Q102289.fctrYes                     
## Q102674.fctrNo                   .  
## Q102674.fctrYes                     
## Q102687.fctrNo                   .  
## Q102687.fctrYes                  *  
## Q102906.fctrNo                      
## Q102906.fctrYes                     
## Q103293.fctrNo                      
## Q103293.fctrYes                     
## Q104996.fctrNo                      
## Q104996.fctrYes                     
## Q105655.fctrNo                      
## Q105655.fctrYes                     
## Q105840.fctrNo                      
## Q105840.fctrYes                     
## Q106042.fctrNo                      
## Q106042.fctrYes                     
## Q106272.fctrNo                      
## Q106272.fctrYes                     
## Q106388.fctrNo                      
## Q106388.fctrYes                     
## Q106389.fctrNo                      
## Q106389.fctrYes                     
## Q106993.fctrNo                      
## Q106993.fctrYes                     
## Q106997.fctrGr                      
## Q106997.fctrYy                      
## Q107491.fctrNo                      
## Q107491.fctrYes                     
## Q107869.fctrNo                      
## Q107869.fctrYes                     
## `Q108342.fctrIn-person`             
## Q108342.fctrOnline               *  
## Q108343.fctrNo                      
## Q108343.fctrYes                     
## Q108617.fctrNo                      
## Q108617.fctrYes                     
## Q108754.fctrNo                      
## Q108754.fctrYes                     
## Q108855.fctrUmm...                  
## `Q108855.fctrYes!`                  
## Q108856.fctrSocialize               
## Q108856.fctrSpace                   
## Q108950.fctrCautious                
## `Q108950.fctrRisk-friendly`         
## Q109244.fctrNo                   ***
## Q109244.fctrYes                  ***
## Q109367.fctrNo                      
## Q109367.fctrYes                     
## Q110740.fctrMac                     
## Q110740.fctrPC                   .  
## Q111220.fctrNo                      
## Q111220.fctrYes                     
## Q111580.fctrDemanding               
## Q111580.fctrSupportive              
## Q111848.fctrNo                      
## Q111848.fctrYes                     
## Q112270.fctrNo                      
## Q112270.fctrYes                     
## Q112478.fctrNo                   *  
## Q112478.fctrYes                     
## Q112512.fctrNo                      
## Q112512.fctrYes                     
## Q113181.fctrNo                      
## Q113181.fctrYes                     
## Q113583.fctrTalk                    
## Q113583.fctrTunes                   
## Q113584.fctrPeople                  
## Q113584.fctrTechnology              
## Q113992.fctrNo                      
## Q113992.fctrYes                  .  
## Q114152.fctrNo                      
## Q114152.fctrYes                     
## Q114386.fctrMysterious              
## Q114386.fctrTMI                     
## Q114517.fctrNo                      
## Q114517.fctrYes                     
## Q114748.fctrNo                   .  
## Q114748.fctrYes                     
## Q114961.fctrNo                      
## Q114961.fctrYes                     
## Q115195.fctrNo                      
## Q115195.fctrYes                     
## Q115390.fctrNo                      
## Q115390.fctrYes                     
## Q115602.fctrNo                      
## Q115602.fctrYes                     
## Q115610.fctrNo                      
## Q115610.fctrYes                     
## Q115611.fctrNo                      
## Q115611.fctrYes                  ** 
## Q115777.fctrEnd                     
## Q115777.fctrStart                   
## Q115899.fctrCs                      
## Q115899.fctrMe                      
## Q116197.fctrA.M.                 *  
## Q116197.fctrP.M.                 .  
## Q116441.fctrNo                      
## Q116441.fctrYes                     
## Q116448.fctrNo                      
## Q116448.fctrYes                     
## Q116601.fctrNo                      
## Q116601.fctrYes                     
## Q116797.fctrNo                      
## Q116797.fctrYes                     
## Q116881.fctrHappy                   
## Q116881.fctrRight                   
## Q116953.fctrNo                      
## Q116953.fctrYes                     
## `Q117186.fctrCool headed`           
## `Q117186.fctrHot headed`            
## `Q117193.fctrOdd hours`             
## `Q117193.fctrStandard hours`        
## Q118117.fctrNo                      
## Q118117.fctrYes                     
## Q118232.fctrId                   ** 
## Q118232.fctrPr                      
## Q118233.fctrNo                      
## Q118233.fctrYes                     
## Q118237.fctrNo                      
## Q118237.fctrYes                     
## Q118892.fctrNo                      
## Q118892.fctrYes                     
## Q119334.fctrNo                      
## Q119334.fctrYes                     
## Q119650.fctrGiving                  
## Q119650.fctrReceiving               
## Q119851.fctrNo                      
## Q119851.fctrYes                     
## Q120012.fctrNo                      
## Q120012.fctrYes                     
## Q120014.fctrNo                      
## Q120014.fctrYes                     
## `Q120194.fctrStudy first`        *  
## `Q120194.fctrTry first`             
## Q120379.fctrNo                      
## Q120379.fctrYes                     
## Q120472.fctrArt                     
## Q120472.fctrScience                 
## Q120650.fctrNo                      
## Q120650.fctrYes                     
## Q120978.fctrNo                      
## Q120978.fctrYes                     
## Q121011.fctrNo                      
## Q121011.fctrYes                     
## Q121699.fctrNo                      
## Q121699.fctrYes                  *  
## Q121700.fctrNo                   .  
## Q121700.fctrYes                     
## Q122120.fctrNo                      
## Q122120.fctrYes                     
## Q122769.fctrNo                      
## Q122769.fctrYes                     
## Q122770.fctrNo                      
## Q122770.fctrYes                     
## Q122771.fctrPc                      
## Q122771.fctrPt                   .  
## Q123464.fctrNo                      
## Q123464.fctrYes                     
## Q123621.fctrNo                      
## Q123621.fctrYes                     
## Q124122.fctrNo                      
## Q124122.fctrYes                     
## Q124742.fctrNo                      
## Q124742.fctrYes                     
## Q96024.fctrNo                       
## Q96024.fctrYes                      
## `Q98059.fctrOnly-child`             
## Q98059.fctrYes                      
## Q98078.fctrNo                       
## Q98078.fctrYes                      
## Q98197.fctrNo                    *  
## Q98197.fctrYes                      
## Q98578.fctrNo                    *  
## Q98578.fctrYes                   .  
## Q98869.fctrNo                    ** 
## Q98869.fctrYes                      
## Q99480.fctrNo                       
## Q99480.fctrYes                      
## Q99581.fctrNo                       
## Q99581.fctrYes                      
## Q99716.fctrNo                       
## Q99716.fctrYes                      
## `Q99982.fctrCheck!`                 
## Q99982.fctrNope                     
## YOB.Age.fctr.L                   ** 
## YOB.Age.fctr.Q                      
## YOB.Age.fctr.C                      
## `YOB.Age.fctr^4`                 *  
## `YOB.Age.fctr^5`                    
## `YOB.Age.fctr^6`                    
## `YOB.Age.fctr^7`                 .  
## `YOB.Age.fctr^8`                 *  
## `Hhold.fctrN:.clusterid.fctr2`      
## `Hhold.fctrMKn:.clusterid.fctr2`    
## `Hhold.fctrMKy:.clusterid.fctr2`    
## `Hhold.fctrPKn:.clusterid.fctr2`    
## `Hhold.fctrPKy:.clusterid.fctr2`    
## `Hhold.fctrSKn:.clusterid.fctr2`    
## `Hhold.fctrSKy:.clusterid.fctr2` *  
## `Hhold.fctrN:.clusterid.fctr3`      
## `Hhold.fctrMKn:.clusterid.fctr3`    
## `Hhold.fctrMKy:.clusterid.fctr3`    
## `Hhold.fctrPKn:.clusterid.fctr3`    
## `Hhold.fctrPKy:.clusterid.fctr3`    
## `Hhold.fctrSKn:.clusterid.fctr3`    
## `Hhold.fctrSKy:.clusterid.fctr3`    
## `Hhold.fctrN:.clusterid.fctr4`      
## `Hhold.fctrMKn:.clusterid.fctr4`    
## `Hhold.fctrMKy:.clusterid.fctr4`    
## `Hhold.fctrPKn:.clusterid.fctr4`    
## `Hhold.fctrPKy:.clusterid.fctr4`    
## `Hhold.fctrSKn:.clusterid.fctr4`    
## `Hhold.fctrSKy:.clusterid.fctr4`    
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## (Dispersion parameter for binomial family taken to be 1)
## 
##     Null deviance: 6150.3  on 4447  degrees of freedom
## Residual deviance: 5311.8  on 4199  degrees of freedom
## AIC: 5809.8
## 
## Number of Fisher Scoring iterations: 5
## 
## [1] "myfit_mdl: train diagnostics complete: 22.697000 secs"

## Warning in predict.lm(object, newdata, se.fit, scale = 1, type =
## ifelse(type == : prediction from a rank-deficient fit may be misleading

## Warning in predict.lm(object, newdata, se.fit, scale = 1, type =
## ifelse(type == : prediction from a rank-deficient fit may be misleading

##          Prediction
## Reference    R    D
##         R 1831  260
##         D 1358  999
##       Accuracy          Kappa  AccuracyLower  AccuracyUpper   AccuracyNull 
##   6.362410e-01   2.908824e-01   6.219049e-01   6.503980e-01   5.299011e-01 
## AccuracyPValue  McnemarPValue 
##   9.324900e-47  9.109506e-164

## Warning in predict.lm(object, newdata, se.fit, scale = 1, type =
## ifelse(type == : prediction from a rank-deficient fit may be misleading

## Warning in predict.lm(object, newdata, se.fit, scale = 1, type =
## ifelse(type == : prediction from a rank-deficient fit may be misleading

##          Prediction
## Reference   R   D
##         R 483  43
##         D 436 158
##       Accuracy          Kappa  AccuracyLower  AccuracyUpper   AccuracyNull 
##   5.723214e-01   1.766880e-01   5.427420e-01   6.015200e-01   5.303571e-01 
## AccuracyPValue  McnemarPValue 
##   2.638913e-03   9.689041e-72 
## [1] "myfit_mdl: predict complete: 33.739000 secs"
##               id
## 1 All.X##rcv#glm
##                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               feats
## 1 Q109244.fctr,Hhold.fctr,Edn.fctr,Q101163.fctr,Q100689.fctr,Q98078.fctr,Q99716.fctr,Q120379.fctr,Q121699.fctr,Q105840.fctr,Q113583.fctr,Q115195.fctr,Q102089.fctr,Q98059.fctr,Q114386.fctr,Q100680.fctr,Q108342.fctr,Q111848.fctr,YOB.Age.fctr,Q118892.fctr,Q102687.fctr,Q115390.fctr,Q119851.fctr,Q114517.fctr,Q120012.fctr,Q109367.fctr,Q114961.fctr,Q121700.fctr,Q124122.fctr,Q111220.fctr,Q113992.fctr,Q121011.fctr,Q106042.fctr,Q116448.fctr,Q116601.fctr,Q104996.fctr,Q102906.fctr,Q113584.fctr,Q108950.fctr,Q102674.fctr,Q103293.fctr,Q112478.fctr,Q114748.fctr,Q107491.fctr,Q100562.fctr,Q108617.fctr,Q100010.fctr,Q115602.fctr,Q116953.fctr,Q115610.fctr,Q106997.fctr,Q120978.fctr,Q112512.fctr,Q108343.fctr,Q96024.fctr,Q106389.fctr,.rnorm,Q108754.fctr,Q98578.fctr,Q101162.fctr,Q115777.fctr,Q99581.fctr,Q124742.fctr,Q116797.fctr,Q112270.fctr,Q118237.fctr,Q119650.fctr,Q111580.fctr,Q123464.fctr,Q117193.fctr,Q99982.fctr,Q108856.fctr,Q118233.fctr,Q102289.fctr,Q116197.fctr,Income.fctr,Q118232.fctr,Q120194.fctr,Q114152.fctr,Q122770.fctr,Q117186.fctr,Q105655.fctr,Q106993.fctr,Q119334.fctr,Q122120.fctr,Q116441.fctr,Q118117.fctr,Q123621.fctr,Q122769.fctr,Q120650.fctr,Q98869.fctr,Q107869.fctr,Q120014.fctr,Q115899.fctr,Q106388.fctr,Q99480.fctr,Q122771.fctr,Q108855.fctr,Q110740.fctr,Q106272.fctr,Q101596.fctr,Q116881.fctr,Q120472.fctr,Q98197.fctr,Q113181.fctr,Q115611.fctr,Gender.fctr,Hhold.fctr:.clusterid.fctr
##   max.nTuningRuns min.elapsedtime.everything min.elapsedtime.final
## 1               1                     19.705                 1.774
##   max.AUCpROC.fit max.Sens.fit max.Spec.fit max.AUCROCR.fit
## 1       0.6685636    0.6451459    0.6919813       0.2601896
##   opt.prob.threshold.fit max.f.score.fit max.Accuracy.fit
## 1                   0.65       0.6935606        0.6029697
##   max.AccuracyLower.fit max.AccuracyUpper.fit max.Kappa.fit
## 1             0.6219049              0.650398     0.2025975
##   max.AUCpROC.OOB max.Sens.OOB max.Spec.OOB max.AUCROCR.OOB
## 1        0.594769    0.5380228    0.6515152       0.3358234
##   opt.prob.threshold.OOB max.f.score.OOB max.Accuracy.OOB
## 1                   0.75       0.6685121        0.5723214
##   max.AccuracyLower.OOB max.AccuracyUpper.OOB max.Kappa.OOB
## 1              0.542742               0.60152      0.176688
##   max.AccuracySD.fit max.KappaSD.fit
## 1        0.008448491      0.01641592
## [1] "myfit_mdl: exit: 33.754000 secs"

# Check if other preProcess methods improve model performance
fit.models_1_chunk_df <- 
    myadd_chunk(fit.models_1_chunk_df, "fit.models_1_preProc", major.inc = FALSE,
                label.minor = "preProc")

##                  label step_major step_minor label_minor     bgn     end
## 4   fit.models_1_All.X          1          3         glm 380.925 414.731
## 5 fit.models_1_preProc          1          4     preProc 414.732      NA
##   elapsed
## 4  33.806
## 5      NA

require(gdata)

## Loading required package: gdata

## gdata: read.xls support for 'XLS' (Excel 97-2004) files ENABLED.

##

## gdata: read.xls support for 'XLSX' (Excel 2007+) files ENABLED.

## 
## Attaching package: 'gdata'

## The following objects are masked from 'package:dplyr':
## 
##     combine, first, last

## The following object is masked from 'package:stats':
## 
##     nobs

## The following object is masked from 'package:utils':
## 
##     object.size

mdl_id <- orderBy(get_model_sel_frmla(), glb_models_df)[1, "id"]
indepVar <- trim(unlist(strsplit(glb_models_df[glb_models_df$id == mdl_id,
                                                      "feats"], "[,]")))
method <- tail(unlist(strsplit(mdl_id, "[.]")), 1)
mdl_id_pfx <- paste0(head(unlist(strsplit(mdl_id, "[.]")), -1), collapse = ".")
if (!is.null(glbObsFitOutliers[[mdl_id_pfx]])) {
    fitobs_df <- glbObsFit[!(glbObsFit[, glbFeatsId] %in%
                                     glbObsFitOutliers[[mdl_id_pfx]]), ]
        print(sprintf("Outliers removed: %d", nrow(glbObsFit) - nrow(fitobs_df)))
        print(setdiff(glbObsFit[, glbFeatsId], fitobs_df[, glbFeatsId]))
    
} else fitobs_df <- glbObsFit

for (prePr in glb_preproc_methods) {   
    # The operations are applied in this order: 
    #   Box-Cox/Yeo-Johnson transformation, centering, scaling, range, imputation, PCA, ICA then spatial sign.
    
    ret_lst <- myfit_mdl(mdl_specs_lst=myinit_mdl_specs_lst(mdl_specs_lst=list(
            id.prefix=mdl_id_pfx, 
            type=glb_model_type, tune.df=glbMdlTuneParams,
            trainControl.method="repeatedcv",
            trainControl.number=glb_rcv_n_folds,
            trainControl.repeats=glb_rcv_n_repeats,
            trainControl.classProbs = glb_is_classification,
            trainControl.summaryFunction = glbMdlMetricSummaryFn,
            train.metric = glbMdlMetricSummary, 
            train.maximize = glbMdlMetricMaximize,    
            train.method=method, train.preProcess=prePr)),
            indepVar=indepVar, rsp_var=glb_rsp_var, 
            fit_df=fitobs_df, OOB_df=glbObsOOB)
}            
    
    # If (All|RFE).X.glm is less accurate than Low.Cor.X.glm
    #   check NA coefficients & filter appropriate terms in indepVar
#     if (method == "glm") {
#         orig_glm <- glb_models_lst[[paste0(mdl_id, ".", model_method)]]$finalModel
#         orig_glm <- glb_models_lst[["All.X.glm"]]$finalModel; print(summary(orig_glm))
#         orig_glm <- glb_models_lst[["RFE.X.glm"]]$finalModel; print(summary(orig_glm))
#           require(car)
#           vif_orig_glm <- vif(orig_glm); print(vif_orig_glm)
#           # if vif errors out with "there are aliased coefficients in the model"
#               alias_orig_glm <- alias(orig_glm); alias_complete_orig_glm <- (alias_orig_glm$Complete > 0); alias_complete_orig_glm <- alias_complete_orig_glm[rowSums(alias_complete_orig_glm) > 0, colSums(alias_complete_orig_glm) > 0]; print(alias_complete_orig_glm)
#           print(vif_orig_glm[!is.na(vif_orig_glm) & (vif_orig_glm == Inf)])
#           print(which.max(vif_orig_glm))
#           print(sort(vif_orig_glm[vif_orig_glm >= 1.0e+03], decreasing=TRUE))
#           glbObsFit[c(1143, 3637, 3953, 4105), c("UniqueID", "Popular", "H.P.quandary", "Headline")]
#           glb_feats_df[glb_feats_df$id %in% grep("[HSA]\\.chrs.n.log", glb_feats_df$id, value=TRUE) | glb_feats_df$cor.high.X %in%    grep("[HSA]\\.chrs.n.log", glb_feats_df$id, value=TRUE), ]
#           all.equal(glbObsAll$S.chrs.uppr.n.log, glbObsAll$A.chrs.uppr.n.log)
#           cor(glbObsAll$S.T.herald, glbObsAll$S.T.tribun)
#           mydspObs(Abstract.contains="[Dd]iar", cols=("Abstract"), all=TRUE)
#           subset(glb_feats_df, cor.y.abs <= glb_feats_df[glb_feats_df$id == ".rnorm", "cor.y.abs"])
#         corxx_mtrx <- cor(data.matrix(glbObsAll[, setdiff(names(glbObsAll), myfind_chr_cols_df(glbObsAll))]), use="pairwise.complete.obs"); abs_corxx_mtrx <- abs(corxx_mtrx); diag(abs_corxx_mtrx) <- 0
#           which.max(abs_corxx_mtrx["S.T.tribun", ])
#           abs_corxx_mtrx["A.npnct08.log", "S.npnct08.log"]
#         step_glm <- step(orig_glm)
#     }
    # Since caret does not optimize rpart well
#     if (method == "rpart")
#         ret_lst <- myfit_mdl(mdl_id=paste0(mdl_id_pfx, ".cp.0"), model_method=method,
#                                 indepVar=indepVar,
#                                 model_type=glb_model_type,
#                                 rsp_var=glb_rsp_var,
#                                 fit_df=glbObsFit, OOB_df=glbObsOOB,        
#             n_cv_folds=0, tune_models_df=data.frame(parameter="cp", min=0.0, max=0.0, by=0.1))

# User specified
#   Ensure at least 2 vars in each regression; else varImp crashes
# sav_models_lst <- glb_models_lst; sav_models_df <- glb_models_df; sav_featsimp_df <- glb_featsimp_df; all.equal(sav_featsimp_df, glb_featsimp_df)
# glb_models_lst <- sav_models_lst; glb_models_df <- sav_models_df; glm_featsimp_df <- sav_featsimp_df

    # easier to exclude features
# require(gdata) # needed for trim
# mdl_id <- "";
# indepVar <- head(subset(glb_models_df, grepl("All\\.X\\.", mdl_id), select=feats)
#                         , 1)[, "feats"]
# indepVar <- trim(unlist(strsplit(indepVar, "[,]")))
# indepVar <- setdiff(indepVar, ".rnorm")

    # easier to include features
#stop(here"); sav_models_df <- glb_models_df; glb_models_df <- sav_models_df
# !_sp
# mdl_id <- "csm"; indepVar <- c(NULL
#     ,"prdline.my.fctr", "prdline.my.fctr:.clusterid.fctr"
#     ,"prdline.my.fctr*biddable"
#     #,"prdline.my.fctr*startprice.log"
#     #,"prdline.my.fctr*startprice.diff"    
#     ,"prdline.my.fctr*condition.fctr"
#     ,"prdline.my.fctr*D.terms.post.stop.n"
#     #,"prdline.my.fctr*D.terms.post.stem.n"
#     ,"prdline.my.fctr*cellular.fctr"    
# #    ,"<feat1>:<feat2>"
#                                            )
# for (method in glbMdlMethods) {
#     ret_lst <- myfit_mdl(mdl_id=mdl_id, model_method=method,
#                                 indepVar=indepVar,
#                                 model_type=glb_model_type,
#                                 rsp_var=glb_rsp_var,
#                                 fit_df=glbObsFit, OOB_df=glbObsOOB,
#                     n_cv_folds=glb_rcv_n_folds, tune_models_df=glbMdlTuneParams)
#     csm_mdl_id <- paste0(mdl_id, ".", method)
#     csm_featsimp_df <- myget_feats_importance(glb_models_lst[[paste0(mdl_id, ".",
#                                                                      method)]]);               print(head(csm_featsimp_df))
# }
###

# Ntv.1.lm <- lm(reformulate(indepVar, glb_rsp_var), glbObsTrn); print(summary(Ntv.1.lm))

#glb_models_df[, "max.Accuracy.OOB", FALSE]
#varImp(glb_models_lst[["Low.cor.X.glm"]])
#orderBy(~ -Overall, varImp(glb_models_lst[["All.X.2.glm"]])$imp)
#orderBy(~ -Overall, varImp(glb_models_lst[["All.X.3.glm"]])$imp)
#glb_feats_df[grepl("npnct28", glb_feats_df$id), ]

    # User specified bivariate models
#     indepVar_lst <- list()
#     for (feat in setdiff(names(glbObsFit), 
#                          union(glb_rsp_var, glbFeatsExclude)))
#         indepVar_lst[["feat"]] <- feat

    # User specified combinatorial models
#     indepVar_lst <- list()
#     combn_mtrx <- combn(c("<feat1_name>", "<feat2_name>", "<featn_name>"), 
#                           <num_feats_to_choose>)
#     for (combn_ix in 1:ncol(combn_mtrx))
#         #print(combn_mtrx[, combn_ix])
#         indepVar_lst[[combn_ix]] <- combn_mtrx[, combn_ix]
    
    # template for myfit_mdl
    #   rf is hard-coded in caret to recognize only Accuracy / Kappa evaluation metrics
    #       only for OOB in trainControl ?
    
#     ret_lst <- myfit_mdl_fn(mdl_id=paste0(mdl_id_pfx, ""), model_method=method,
#                             indepVar=indepVar,
#                             rsp_var=glb_rsp_var,
#                             fit_df=glbObsFit, OOB_df=glbObsOOB,
#                             n_cv_folds=glb_rcv_n_folds, tune_models_df=glbMdlTuneParams,
#                             model_loss_mtrx=glbMdlMetric_terms,
#                             model_summaryFunction=glbMdlMetricSummaryFn,
#                             model_metric=glbMdlMetricSummary,
#                             model_metric_maximize=glbMdlMetricMaximize)

# Simplify a model
# fit_df <- glbObsFit; glb_mdl <- step(<complex>_mdl)

# Non-caret models
#     rpart_area_mdl <- rpart(reformulate("Area", response=glb_rsp_var), 
#                                data=glbObsFit, #method="class", 
#                                control=rpart.control(cp=0.12),
#                            parms=list(loss=glbMdlMetric_terms))
#     print("rpart_sel_wlm_mdl"); prp(rpart_sel_wlm_mdl)
# 

print(glb_models_df)

##                                                              id
## MFO###myMFO_classfr                         MFO###myMFO_classfr
## Random###myrandom_classfr             Random###myrandom_classfr
## Max.cor.Y.rcv.1X1###glmnet           Max.cor.Y.rcv.1X1###glmnet
## Max.cor.Y##rcv#rpart                       Max.cor.Y##rcv#rpart
## Interact.High.cor.Y##rcv#glmnet Interact.High.cor.Y##rcv#glmnet
## Low.cor.X##rcv#glmnet                     Low.cor.X##rcv#glmnet
## All.X##rcv#glmnet                             All.X##rcv#glmnet
## All.X##rcv#glm                                   All.X##rcv#glm
##                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             feats
## MFO###myMFO_classfr                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        .rnorm
## Random###myrandom_classfr                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  .rnorm
## Max.cor.Y.rcv.1X1###glmnet                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               Q109244.fctr,Gender.fctr
## Max.cor.Y##rcv#rpart                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     Q109244.fctr,Gender.fctr
## Interact.High.cor.Y##rcv#glmnet                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            Q109244.fctr,Gender.fctr,Q109244.fctr:Q99480.fctr,Q109244.fctr:Q98078.fctr,Q109244.fctr:Q100689.fctr,Q109244.fctr:Q108855.fctr,Q109244.fctr:Q123621.fctr,Q109244.fctr:Q122771.fctr,Q109244.fctr:Q120472.fctr,Q109244.fctr:Q106272.fctr
## Low.cor.X##rcv#glmnet           Q109244.fctr,Hhold.fctr,Edn.fctr,Q101163.fctr,Q100689.fctr,Q98078.fctr,Q99716.fctr,Q120379.fctr,Q121699.fctr,Q105840.fctr,Q113583.fctr,Q115195.fctr,Q102089.fctr,Q98059.fctr,Q114386.fctr,Q100680.fctr,Q108342.fctr,Q111848.fctr,YOB.Age.fctr,Q118892.fctr,Q102687.fctr,Q115390.fctr,Q119851.fctr,Q114517.fctr,Q120012.fctr,Q109367.fctr,Q114961.fctr,Q121700.fctr,Q124122.fctr,Q111220.fctr,Q113992.fctr,Q121011.fctr,Q106042.fctr,Q116448.fctr,Q116601.fctr,Q104996.fctr,Q102906.fctr,Q113584.fctr,Q108950.fctr,Q102674.fctr,Q103293.fctr,Q112478.fctr,Q114748.fctr,Q107491.fctr,Q100562.fctr,Q108617.fctr,Q100010.fctr,Q115602.fctr,Q116953.fctr,Q115610.fctr,Q106997.fctr,Q120978.fctr,Q112512.fctr,Q108343.fctr,Q96024.fctr,Q106389.fctr,.rnorm,Q108754.fctr,Q98578.fctr,Q101162.fctr,Q115777.fctr,Q99581.fctr,Q124742.fctr,Q116797.fctr,Q112270.fctr,Q118237.fctr,Q119650.fctr,Q111580.fctr,Q123464.fctr,Q117193.fctr,Q99982.fctr,Q108856.fctr,Q118233.fctr,Q102289.fctr,Q116197.fctr,Income.fctr,Q118232.fctr,Q120194.fctr,Q114152.fctr,Q122770.fctr,Q117186.fctr,Q105655.fctr,Q106993.fctr,Q119334.fctr,Q122120.fctr,Q116441.fctr,Q118117.fctr,Q123621.fctr,Q122769.fctr,Q120650.fctr,Q98869.fctr,Q107869.fctr,Q120014.fctr,Q115899.fctr,Q106388.fctr,Q99480.fctr,Q122771.fctr,Q108855.fctr,Q110740.fctr,Q106272.fctr,Q101596.fctr,Q116881.fctr,Q120472.fctr,Q98197.fctr,Q113181.fctr,Q115611.fctr,Gender.fctr,Hhold.fctr:.clusterid.fctr
## All.X##rcv#glmnet               Q109244.fctr,Hhold.fctr,Edn.fctr,Q101163.fctr,Q100689.fctr,Q98078.fctr,Q99716.fctr,Q120379.fctr,Q121699.fctr,Q105840.fctr,Q113583.fctr,Q115195.fctr,Q102089.fctr,Q98059.fctr,Q114386.fctr,Q100680.fctr,Q108342.fctr,Q111848.fctr,YOB.Age.fctr,Q118892.fctr,Q102687.fctr,Q115390.fctr,Q119851.fctr,Q114517.fctr,Q120012.fctr,Q109367.fctr,Q114961.fctr,Q121700.fctr,Q124122.fctr,Q111220.fctr,Q113992.fctr,Q121011.fctr,Q106042.fctr,Q116448.fctr,Q116601.fctr,Q104996.fctr,Q102906.fctr,Q113584.fctr,Q108950.fctr,Q102674.fctr,Q103293.fctr,Q112478.fctr,Q114748.fctr,Q107491.fctr,Q100562.fctr,Q108617.fctr,Q100010.fctr,Q115602.fctr,Q116953.fctr,Q115610.fctr,Q106997.fctr,Q120978.fctr,Q112512.fctr,Q108343.fctr,Q96024.fctr,Q106389.fctr,.rnorm,Q108754.fctr,Q98578.fctr,Q101162.fctr,Q115777.fctr,Q99581.fctr,Q124742.fctr,Q116797.fctr,Q112270.fctr,Q118237.fctr,Q119650.fctr,Q111580.fctr,Q123464.fctr,Q117193.fctr,Q99982.fctr,Q108856.fctr,Q118233.fctr,Q102289.fctr,Q116197.fctr,Income.fctr,Q118232.fctr,Q120194.fctr,Q114152.fctr,Q122770.fctr,Q117186.fctr,Q105655.fctr,Q106993.fctr,Q119334.fctr,Q122120.fctr,Q116441.fctr,Q118117.fctr,Q123621.fctr,Q122769.fctr,Q120650.fctr,Q98869.fctr,Q107869.fctr,Q120014.fctr,Q115899.fctr,Q106388.fctr,Q99480.fctr,Q122771.fctr,Q108855.fctr,Q110740.fctr,Q106272.fctr,Q101596.fctr,Q116881.fctr,Q120472.fctr,Q98197.fctr,Q113181.fctr,Q115611.fctr,Gender.fctr,Hhold.fctr:.clusterid.fctr
## All.X##rcv#glm                  Q109244.fctr,Hhold.fctr,Edn.fctr,Q101163.fctr,Q100689.fctr,Q98078.fctr,Q99716.fctr,Q120379.fctr,Q121699.fctr,Q105840.fctr,Q113583.fctr,Q115195.fctr,Q102089.fctr,Q98059.fctr,Q114386.fctr,Q100680.fctr,Q108342.fctr,Q111848.fctr,YOB.Age.fctr,Q118892.fctr,Q102687.fctr,Q115390.fctr,Q119851.fctr,Q114517.fctr,Q120012.fctr,Q109367.fctr,Q114961.fctr,Q121700.fctr,Q124122.fctr,Q111220.fctr,Q113992.fctr,Q121011.fctr,Q106042.fctr,Q116448.fctr,Q116601.fctr,Q104996.fctr,Q102906.fctr,Q113584.fctr,Q108950.fctr,Q102674.fctr,Q103293.fctr,Q112478.fctr,Q114748.fctr,Q107491.fctr,Q100562.fctr,Q108617.fctr,Q100010.fctr,Q115602.fctr,Q116953.fctr,Q115610.fctr,Q106997.fctr,Q120978.fctr,Q112512.fctr,Q108343.fctr,Q96024.fctr,Q106389.fctr,.rnorm,Q108754.fctr,Q98578.fctr,Q101162.fctr,Q115777.fctr,Q99581.fctr,Q124742.fctr,Q116797.fctr,Q112270.fctr,Q118237.fctr,Q119650.fctr,Q111580.fctr,Q123464.fctr,Q117193.fctr,Q99982.fctr,Q108856.fctr,Q118233.fctr,Q102289.fctr,Q116197.fctr,Income.fctr,Q118232.fctr,Q120194.fctr,Q114152.fctr,Q122770.fctr,Q117186.fctr,Q105655.fctr,Q106993.fctr,Q119334.fctr,Q122120.fctr,Q116441.fctr,Q118117.fctr,Q123621.fctr,Q122769.fctr,Q120650.fctr,Q98869.fctr,Q107869.fctr,Q120014.fctr,Q115899.fctr,Q106388.fctr,Q99480.fctr,Q122771.fctr,Q108855.fctr,Q110740.fctr,Q106272.fctr,Q101596.fctr,Q116881.fctr,Q120472.fctr,Q98197.fctr,Q113181.fctr,Q115611.fctr,Gender.fctr,Hhold.fctr:.clusterid.fctr
##                                 max.nTuningRuns min.elapsedtime.everything
## MFO###myMFO_classfr                           0                      0.498
## Random###myrandom_classfr                     0                      0.290
## Max.cor.Y.rcv.1X1###glmnet                    0                      0.827
## Max.cor.Y##rcv#rpart                          5                      1.660
## Interact.High.cor.Y##rcv#glmnet              25                      5.291
## Low.cor.X##rcv#glmnet                        25                     24.392
## All.X##rcv#glmnet                            25                     23.189
## All.X##rcv#glm                                1                     19.705
##                                 min.elapsedtime.final max.AUCpROC.fit
## MFO###myMFO_classfr                             0.004       0.5000000
## Random###myrandom_classfr                       0.003       0.4942483
## Max.cor.Y.rcv.1X1###glmnet                      0.064       0.5971118
## Max.cor.Y##rcv#rpart                            0.020       0.5971118
## Interact.High.cor.Y##rcv#glmnet                 0.358       0.6184781
## Low.cor.X##rcv#glmnet                           2.084       0.6273089
## All.X##rcv#glmnet                               2.025       0.6273089
## All.X##rcv#glm                                  1.774       0.6685636
##                                 max.Sens.fit max.Spec.fit max.AUCROCR.fit
## MFO###myMFO_classfr                0.0000000    1.0000000       0.5000000
## Random###myrandom_classfr          0.4619799    0.5265168       0.5073101
## Max.cor.Y.rcv.1X1###glmnet         0.5480631    0.6461604       0.3580613
## Max.cor.Y##rcv#rpart               0.5480631    0.6461604       0.3676308
## Interact.High.cor.Y##rcv#glmnet    0.5958871    0.6410692       0.3319465
## Low.cor.X##rcv#glmnet              0.5155428    0.7390751       0.3048695
## All.X##rcv#glmnet                  0.5155428    0.7390751       0.3048695
## All.X##rcv#glm                     0.6451459    0.6919813       0.2601896
##                                 opt.prob.threshold.fit max.f.score.fit
## MFO###myMFO_classfr                               0.50       0.6395473
## Random###myrandom_classfr                         0.55       0.6395473
## Max.cor.Y.rcv.1X1###glmnet                        0.60       0.6720662
## Max.cor.Y##rcv#rpart                              0.55       0.6720662
## Interact.High.cor.Y##rcv#glmnet                   0.65       0.6727114
## Low.cor.X##rcv#glmnet                             0.60       0.6756231
## All.X##rcv#glmnet                                 0.60       0.6756231
## All.X##rcv#glm                                    0.65       0.6935606
##                                 max.Accuracy.fit max.AccuracyLower.fit
## MFO###myMFO_classfr                    0.4700989             0.4553427
## Random###myrandom_classfr              0.4700989             0.4553427
## Max.cor.Y.rcv.1X1###glmnet             0.5721673             0.5574714
## Max.cor.Y##rcv#rpart                   0.6000450             0.5574714
## Interact.High.cor.Y##rcv#glmnet        0.6058167             0.5633390
## Low.cor.X##rcv#glmnet                  0.6241754             0.5728218
## All.X##rcv#glmnet                      0.6241754             0.5728218
## All.X##rcv#glm                         0.6029697             0.6219049
##                                 max.AccuracyUpper.fit max.Kappa.fit
## MFO###myMFO_classfr                         0.4848945     0.0000000
## Random###myrandom_classfr                   0.4848945     0.0000000
## Max.cor.Y.rcv.1X1###glmnet                  0.5867683     0.1772539
## Max.cor.Y##rcv#rpart                        0.5867683     0.1947896
## Interact.High.cor.Y##rcv#glmnet             0.5925837     0.2088694
## Low.cor.X##rcv#glmnet                       0.6019734     0.2388455
## All.X##rcv#glmnet                           0.6019734     0.2388455
## All.X##rcv#glm                              0.6503980     0.2025975
##                                 max.AUCpROC.OOB max.Sens.OOB max.Spec.OOB
## MFO###myMFO_classfr                   0.5000000    0.0000000    1.0000000
## Random###myrandom_classfr             0.5235690    0.5000000    0.5471380
## Max.cor.Y.rcv.1X1###glmnet            0.5896897    0.5228137    0.6565657
## Max.cor.Y##rcv#rpart                  0.5896897    0.5228137    0.6565657
## Interact.High.cor.Y##rcv#glmnet       0.6031353    0.5665399    0.6397306
## Low.cor.X##rcv#glmnet                 0.6237374    0.5000000    0.7474747
## All.X##rcv#glmnet                     0.6237374    0.5000000    0.7474747
## All.X##rcv#glm                        0.5947690    0.5380228    0.6515152
##                                 max.AUCROCR.OOB opt.prob.threshold.OOB
## MFO###myMFO_classfr                   0.5000000                   0.50
## Random###myrandom_classfr             0.5191202                   0.55
## Max.cor.Y.rcv.1X1###glmnet            0.3658672                   0.60
## Max.cor.Y##rcv#rpart                  0.3774772                   0.55
## Interact.High.cor.Y##rcv#glmnet       0.3571392                   0.65
## Low.cor.X##rcv#glmnet                 0.3184875                   0.70
## All.X##rcv#glmnet                     0.3184875                   0.70
## All.X##rcv#glm                        0.3358234                   0.75
##                                 max.f.score.OOB max.Accuracy.OOB
## MFO###myMFO_classfr                   0.6391252        0.4696429
## Random###myrandom_classfr             0.6391252        0.4696429
## Max.cor.Y.rcv.1X1###glmnet            0.6643789        0.5633929
## Max.cor.Y##rcv#rpart                  0.6643789        0.5633929
## Interact.High.cor.Y##rcv#glmnet       0.6680556        0.5732143
## Low.cor.X##rcv#glmnet                 0.6644250        0.5544643
## All.X##rcv#glmnet                     0.6644250        0.5544643
## All.X##rcv#glm                        0.6685121        0.5723214
##                                 max.AccuracyLower.OOB
## MFO###myMFO_classfr                         0.4400805
## Random###myrandom_classfr                   0.4400805
## Max.cor.Y.rcv.1X1###glmnet                  0.5337655
## Max.cor.Y##rcv#rpart                        0.5337655
## Interact.High.cor.Y##rcv#glmnet             0.5436402
## Low.cor.X##rcv#glmnet                       0.5247985
## All.X##rcv#glmnet                           0.5247985
## All.X##rcv#glm                              0.5427420
##                                 max.AccuracyUpper.OOB max.Kappa.OOB
## MFO###myMFO_classfr                         0.4993651     0.0000000
## Random###myrandom_classfr                   0.4993651     0.0000000
## Max.cor.Y.rcv.1X1###glmnet                  0.5926864     0.1605510
## Max.cor.Y##rcv#rpart                        0.5926864     0.1605510
## Interact.High.cor.Y##rcv#glmnet             0.6024028     0.1779779
## Low.cor.X##rcv#glmnet                       0.5838432     0.1460545
## All.X##rcv#glmnet                           0.5838432     0.1460545
## All.X##rcv#glm                              0.6015200     0.1766880
##                                 max.AccuracySD.fit max.KappaSD.fit
## MFO###myMFO_classfr                             NA              NA
## Random###myrandom_classfr                       NA              NA
## Max.cor.Y.rcv.1X1###glmnet                      NA              NA
## Max.cor.Y##rcv#rpart                   0.012403504      0.02559319
## Interact.High.cor.Y##rcv#glmnet        0.013121299      0.02732571
## Low.cor.X##rcv#glmnet                  0.004938833      0.01189291
## All.X##rcv#glmnet                      0.004938833      0.01189291
## All.X##rcv#glm                         0.008448491      0.01641592

rm(ret_lst)
fit.models_1_chunk_df <- 
    myadd_chunk(fit.models_1_chunk_df, "fit.models_1_end", major.inc = FALSE,
                label.minor = "teardown")

##                  label step_major step_minor label_minor     bgn     end
## 5 fit.models_1_preProc          1          4     preProc 414.732 415.945
## 6     fit.models_1_end          1          5    teardown 415.946      NA
##   elapsed
## 5   1.213
## 6      NA

glb_chunks_df <- myadd_chunk(glb_chunks_df, "fit.models", major.inc = FALSE)

##        label step_major step_minor label_minor     bgn     end elapsed
## 5 fit.models          4          1           1 341.558 415.955  74.397
## 6 fit.models          4          2           2 415.955      NA      NA

fit.models_2_chunk_df <- 
    myadd_chunk(NULL, "fit.models_2_bgn", label.minor = "setup")

##              label step_major step_minor label_minor     bgn end elapsed
## 1 fit.models_2_bgn          1          0       setup 420.029  NA      NA

plt_models_df <- glb_models_df[, -grep("SD|Upper|Lower", names(glb_models_df))]
for (var in grep("^min.", names(plt_models_df), value=TRUE)) {
    plt_models_df[, sub("min.", "inv.", var)] <- 
        #ifelse(all(is.na(tmp <- plt_models_df[, var])), NA, 1.0 / tmp)
        1.0 / plt_models_df[, var]
    plt_models_df <- plt_models_df[ , -grep(var, names(plt_models_df))]
}
print(plt_models_df)

##                                                              id
## MFO###myMFO_classfr                         MFO###myMFO_classfr
## Random###myrandom_classfr             Random###myrandom_classfr
## Max.cor.Y.rcv.1X1###glmnet           Max.cor.Y.rcv.1X1###glmnet
## Max.cor.Y##rcv#rpart                       Max.cor.Y##rcv#rpart
## Interact.High.cor.Y##rcv#glmnet Interact.High.cor.Y##rcv#glmnet
## Low.cor.X##rcv#glmnet                     Low.cor.X##rcv#glmnet
## All.X##rcv#glmnet                             All.X##rcv#glmnet
## All.X##rcv#glm                                   All.X##rcv#glm
##                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             feats
## MFO###myMFO_classfr                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        .rnorm
## Random###myrandom_classfr                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  .rnorm
## Max.cor.Y.rcv.1X1###glmnet                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               Q109244.fctr,Gender.fctr
## Max.cor.Y##rcv#rpart                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     Q109244.fctr,Gender.fctr
## Interact.High.cor.Y##rcv#glmnet                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            Q109244.fctr,Gender.fctr,Q109244.fctr:Q99480.fctr,Q109244.fctr:Q98078.fctr,Q109244.fctr:Q100689.fctr,Q109244.fctr:Q108855.fctr,Q109244.fctr:Q123621.fctr,Q109244.fctr:Q122771.fctr,Q109244.fctr:Q120472.fctr,Q109244.fctr:Q106272.fctr
## Low.cor.X##rcv#glmnet           Q109244.fctr,Hhold.fctr,Edn.fctr,Q101163.fctr,Q100689.fctr,Q98078.fctr,Q99716.fctr,Q120379.fctr,Q121699.fctr,Q105840.fctr,Q113583.fctr,Q115195.fctr,Q102089.fctr,Q98059.fctr,Q114386.fctr,Q100680.fctr,Q108342.fctr,Q111848.fctr,YOB.Age.fctr,Q118892.fctr,Q102687.fctr,Q115390.fctr,Q119851.fctr,Q114517.fctr,Q120012.fctr,Q109367.fctr,Q114961.fctr,Q121700.fctr,Q124122.fctr,Q111220.fctr,Q113992.fctr,Q121011.fctr,Q106042.fctr,Q116448.fctr,Q116601.fctr,Q104996.fctr,Q102906.fctr,Q113584.fctr,Q108950.fctr,Q102674.fctr,Q103293.fctr,Q112478.fctr,Q114748.fctr,Q107491.fctr,Q100562.fctr,Q108617.fctr,Q100010.fctr,Q115602.fctr,Q116953.fctr,Q115610.fctr,Q106997.fctr,Q120978.fctr,Q112512.fctr,Q108343.fctr,Q96024.fctr,Q106389.fctr,.rnorm,Q108754.fctr,Q98578.fctr,Q101162.fctr,Q115777.fctr,Q99581.fctr,Q124742.fctr,Q116797.fctr,Q112270.fctr,Q118237.fctr,Q119650.fctr,Q111580.fctr,Q123464.fctr,Q117193.fctr,Q99982.fctr,Q108856.fctr,Q118233.fctr,Q102289.fctr,Q116197.fctr,Income.fctr,Q118232.fctr,Q120194.fctr,Q114152.fctr,Q122770.fctr,Q117186.fctr,Q105655.fctr,Q106993.fctr,Q119334.fctr,Q122120.fctr,Q116441.fctr,Q118117.fctr,Q123621.fctr,Q122769.fctr,Q120650.fctr,Q98869.fctr,Q107869.fctr,Q120014.fctr,Q115899.fctr,Q106388.fctr,Q99480.fctr,Q122771.fctr,Q108855.fctr,Q110740.fctr,Q106272.fctr,Q101596.fctr,Q116881.fctr,Q120472.fctr,Q98197.fctr,Q113181.fctr,Q115611.fctr,Gender.fctr,Hhold.fctr:.clusterid.fctr
## All.X##rcv#glmnet               Q109244.fctr,Hhold.fctr,Edn.fctr,Q101163.fctr,Q100689.fctr,Q98078.fctr,Q99716.fctr,Q120379.fctr,Q121699.fctr,Q105840.fctr,Q113583.fctr,Q115195.fctr,Q102089.fctr,Q98059.fctr,Q114386.fctr,Q100680.fctr,Q108342.fctr,Q111848.fctr,YOB.Age.fctr,Q118892.fctr,Q102687.fctr,Q115390.fctr,Q119851.fctr,Q114517.fctr,Q120012.fctr,Q109367.fctr,Q114961.fctr,Q121700.fctr,Q124122.fctr,Q111220.fctr,Q113992.fctr,Q121011.fctr,Q106042.fctr,Q116448.fctr,Q116601.fctr,Q104996.fctr,Q102906.fctr,Q113584.fctr,Q108950.fctr,Q102674.fctr,Q103293.fctr,Q112478.fctr,Q114748.fctr,Q107491.fctr,Q100562.fctr,Q108617.fctr,Q100010.fctr,Q115602.fctr,Q116953.fctr,Q115610.fctr,Q106997.fctr,Q120978.fctr,Q112512.fctr,Q108343.fctr,Q96024.fctr,Q106389.fctr,.rnorm,Q108754.fctr,Q98578.fctr,Q101162.fctr,Q115777.fctr,Q99581.fctr,Q124742.fctr,Q116797.fctr,Q112270.fctr,Q118237.fctr,Q119650.fctr,Q111580.fctr,Q123464.fctr,Q117193.fctr,Q99982.fctr,Q108856.fctr,Q118233.fctr,Q102289.fctr,Q116197.fctr,Income.fctr,Q118232.fctr,Q120194.fctr,Q114152.fctr,Q122770.fctr,Q117186.fctr,Q105655.fctr,Q106993.fctr,Q119334.fctr,Q122120.fctr,Q116441.fctr,Q118117.fctr,Q123621.fctr,Q122769.fctr,Q120650.fctr,Q98869.fctr,Q107869.fctr,Q120014.fctr,Q115899.fctr,Q106388.fctr,Q99480.fctr,Q122771.fctr,Q108855.fctr,Q110740.fctr,Q106272.fctr,Q101596.fctr,Q116881.fctr,Q120472.fctr,Q98197.fctr,Q113181.fctr,Q115611.fctr,Gender.fctr,Hhold.fctr:.clusterid.fctr
## All.X##rcv#glm                  Q109244.fctr,Hhold.fctr,Edn.fctr,Q101163.fctr,Q100689.fctr,Q98078.fctr,Q99716.fctr,Q120379.fctr,Q121699.fctr,Q105840.fctr,Q113583.fctr,Q115195.fctr,Q102089.fctr,Q98059.fctr,Q114386.fctr,Q100680.fctr,Q108342.fctr,Q111848.fctr,YOB.Age.fctr,Q118892.fctr,Q102687.fctr,Q115390.fctr,Q119851.fctr,Q114517.fctr,Q120012.fctr,Q109367.fctr,Q114961.fctr,Q121700.fctr,Q124122.fctr,Q111220.fctr,Q113992.fctr,Q121011.fctr,Q106042.fctr,Q116448.fctr,Q116601.fctr,Q104996.fctr,Q102906.fctr,Q113584.fctr,Q108950.fctr,Q102674.fctr,Q103293.fctr,Q112478.fctr,Q114748.fctr,Q107491.fctr,Q100562.fctr,Q108617.fctr,Q100010.fctr,Q115602.fctr,Q116953.fctr,Q115610.fctr,Q106997.fctr,Q120978.fctr,Q112512.fctr,Q108343.fctr,Q96024.fctr,Q106389.fctr,.rnorm,Q108754.fctr,Q98578.fctr,Q101162.fctr,Q115777.fctr,Q99581.fctr,Q124742.fctr,Q116797.fctr,Q112270.fctr,Q118237.fctr,Q119650.fctr,Q111580.fctr,Q123464.fctr,Q117193.fctr,Q99982.fctr,Q108856.fctr,Q118233.fctr,Q102289.fctr,Q116197.fctr,Income.fctr,Q118232.fctr,Q120194.fctr,Q114152.fctr,Q122770.fctr,Q117186.fctr,Q105655.fctr,Q106993.fctr,Q119334.fctr,Q122120.fctr,Q116441.fctr,Q118117.fctr,Q123621.fctr,Q122769.fctr,Q120650.fctr,Q98869.fctr,Q107869.fctr,Q120014.fctr,Q115899.fctr,Q106388.fctr,Q99480.fctr,Q122771.fctr,Q108855.fctr,Q110740.fctr,Q106272.fctr,Q101596.fctr,Q116881.fctr,Q120472.fctr,Q98197.fctr,Q113181.fctr,Q115611.fctr,Gender.fctr,Hhold.fctr:.clusterid.fctr
##                                 max.nTuningRuns max.AUCpROC.fit
## MFO###myMFO_classfr                           0       0.5000000
## Random###myrandom_classfr                     0       0.4942483
## Max.cor.Y.rcv.1X1###glmnet                    0       0.5971118
## Max.cor.Y##rcv#rpart                          5       0.5971118
## Interact.High.cor.Y##rcv#glmnet              25       0.6184781
## Low.cor.X##rcv#glmnet                        25       0.6273089
## All.X##rcv#glmnet                            25       0.6273089
## All.X##rcv#glm                                1       0.6685636
##                                 max.Sens.fit max.Spec.fit max.AUCROCR.fit
## MFO###myMFO_classfr                0.0000000    1.0000000       0.5000000
## Random###myrandom_classfr          0.4619799    0.5265168       0.5073101
## Max.cor.Y.rcv.1X1###glmnet         0.5480631    0.6461604       0.3580613
## Max.cor.Y##rcv#rpart               0.5480631    0.6461604       0.3676308
## Interact.High.cor.Y##rcv#glmnet    0.5958871    0.6410692       0.3319465
## Low.cor.X##rcv#glmnet              0.5155428    0.7390751       0.3048695
## All.X##rcv#glmnet                  0.5155428    0.7390751       0.3048695
## All.X##rcv#glm                     0.6451459    0.6919813       0.2601896
##                                 opt.prob.threshold.fit max.f.score.fit
## MFO###myMFO_classfr                               0.50       0.6395473
## Random###myrandom_classfr                         0.55       0.6395473
## Max.cor.Y.rcv.1X1###glmnet                        0.60       0.6720662
## Max.cor.Y##rcv#rpart                              0.55       0.6720662
## Interact.High.cor.Y##rcv#glmnet                   0.65       0.6727114
## Low.cor.X##rcv#glmnet                             0.60       0.6756231
## All.X##rcv#glmnet                                 0.60       0.6756231
## All.X##rcv#glm                                    0.65       0.6935606
##                                 max.Accuracy.fit max.Kappa.fit
## MFO###myMFO_classfr                    0.4700989     0.0000000
## Random###myrandom_classfr              0.4700989     0.0000000
## Max.cor.Y.rcv.1X1###glmnet             0.5721673     0.1772539
## Max.cor.Y##rcv#rpart                   0.6000450     0.1947896
## Interact.High.cor.Y##rcv#glmnet        0.6058167     0.2088694
## Low.cor.X##rcv#glmnet                  0.6241754     0.2388455
## All.X##rcv#glmnet                      0.6241754     0.2388455
## All.X##rcv#glm                         0.6029697     0.2025975
##                                 max.AUCpROC.OOB max.Sens.OOB max.Spec.OOB
## MFO###myMFO_classfr                   0.5000000    0.0000000    1.0000000
## Random###myrandom_classfr             0.5235690    0.5000000    0.5471380
## Max.cor.Y.rcv.1X1###glmnet            0.5896897    0.5228137    0.6565657
## Max.cor.Y##rcv#rpart                  0.5896897    0.5228137    0.6565657
## Interact.High.cor.Y##rcv#glmnet       0.6031353    0.5665399    0.6397306
## Low.cor.X##rcv#glmnet                 0.6237374    0.5000000    0.7474747
## All.X##rcv#glmnet                     0.6237374    0.5000000    0.7474747
## All.X##rcv#glm                        0.5947690    0.5380228    0.6515152
##                                 max.AUCROCR.OOB opt.prob.threshold.OOB
## MFO###myMFO_classfr                   0.5000000                   0.50
## Random###myrandom_classfr             0.5191202                   0.55
## Max.cor.Y.rcv.1X1###glmnet            0.3658672                   0.60
## Max.cor.Y##rcv#rpart                  0.3774772                   0.55
## Interact.High.cor.Y##rcv#glmnet       0.3571392                   0.65
## Low.cor.X##rcv#glmnet                 0.3184875                   0.70
## All.X##rcv#glmnet                     0.3184875                   0.70
## All.X##rcv#glm                        0.3358234                   0.75
##                                 max.f.score.OOB max.Accuracy.OOB
## MFO###myMFO_classfr                   0.6391252        0.4696429
## Random###myrandom_classfr             0.6391252        0.4696429
## Max.cor.Y.rcv.1X1###glmnet            0.6643789        0.5633929
## Max.cor.Y##rcv#rpart                  0.6643789        0.5633929
## Interact.High.cor.Y##rcv#glmnet       0.6680556        0.5732143
## Low.cor.X##rcv#glmnet                 0.6644250        0.5544643
## All.X##rcv#glmnet                     0.6644250        0.5544643
## All.X##rcv#glm                        0.6685121        0.5723214
##                                 max.Kappa.OOB inv.elapsedtime.everything
## MFO###myMFO_classfr                 0.0000000                 2.00803213
## Random###myrandom_classfr           0.0000000                 3.44827586
## Max.cor.Y.rcv.1X1###glmnet          0.1605510                 1.20918984
## Max.cor.Y##rcv#rpart                0.1605510                 0.60240964
## Interact.High.cor.Y##rcv#glmnet     0.1779779                 0.18900019
## Low.cor.X##rcv#glmnet               0.1460545                 0.04099705
## All.X##rcv#glmnet                   0.1460545                 0.04312389
## All.X##rcv#glm                      0.1766880                 0.05074854
##                                 inv.elapsedtime.final
## MFO###myMFO_classfr                       250.0000000
## Random###myrandom_classfr                 333.3333333
## Max.cor.Y.rcv.1X1###glmnet                 15.6250000
## Max.cor.Y##rcv#rpart                       50.0000000
## Interact.High.cor.Y##rcv#glmnet             2.7932961
## Low.cor.X##rcv#glmnet                       0.4798464
## All.X##rcv#glmnet                           0.4938272
## All.X##rcv#glm                              0.5636979

# print(myplot_radar(radar_inp_df=plt_models_df))
# print(myplot_radar(radar_inp_df=subset(plt_models_df, 
#         !(mdl_id %in% grep("random|MFO", plt_models_df$id, value=TRUE)))))

# Compute CI for <metric>SD
glb_models_df <- mutate(glb_models_df, 
                max.df = ifelse(max.nTuningRuns > 1, max.nTuningRuns - 1, NA),
                min.sd2ci.scaler = ifelse(is.na(max.df), NA, qt(0.975, max.df)))
for (var in grep("SD", names(glb_models_df), value=TRUE)) {
    # Does CI alredy exist ?
    var_components <- unlist(strsplit(var, "SD"))
    varActul <- paste0(var_components[1],          var_components[2])
    varUpper <- paste0(var_components[1], "Upper", var_components[2])
    varLower <- paste0(var_components[1], "Lower", var_components[2])
    if (varUpper %in% names(glb_models_df)) {
        warning(varUpper, " already exists in glb_models_df")
        # Assuming Lower also exists
        next
    }    
    print(sprintf("var:%s", var))
    # CI is dependent on sample size in t distribution; df=n-1
    glb_models_df[, varUpper] <- glb_models_df[, varActul] + 
        glb_models_df[, "min.sd2ci.scaler"] * glb_models_df[, var]
    glb_models_df[, varLower] <- glb_models_df[, varActul] - 
        glb_models_df[, "min.sd2ci.scaler"] * glb_models_df[, var]
}

## Warning: max.AccuracyUpper.fit already exists in glb_models_df

## [1] "var:max.KappaSD.fit"

# Plot metrics with CI
plt_models_df <- glb_models_df[, "id", FALSE]
pltCI_models_df <- glb_models_df[, "id", FALSE]
for (var in grep("Upper", names(glb_models_df), value=TRUE)) {
    var_components <- unlist(strsplit(var, "Upper"))
    col_name <- unlist(paste(var_components, collapse=""))
    plt_models_df[, col_name] <- glb_models_df[, col_name]
    for (name in paste0(var_components[1], c("Upper", "Lower"), var_components[2]))
        pltCI_models_df[, name] <- glb_models_df[, name]
}

build_statsCI_data <- function(plt_models_df) {
    mltd_models_df <- melt(plt_models_df, id.vars="id")
    mltd_models_df$data <- sapply(1:nrow(mltd_models_df), 
        function(row_ix) tail(unlist(strsplit(as.character(
            mltd_models_df[row_ix, "variable"]), "[.]")), 1))
    mltd_models_df$label <- sapply(1:nrow(mltd_models_df), 
        function(row_ix) head(unlist(strsplit(as.character(
            mltd_models_df[row_ix, "variable"]), 
            paste0(".", mltd_models_df[row_ix, "data"]))), 1))
    #print(mltd_models_df)
    
    return(mltd_models_df)
}
mltd_models_df <- build_statsCI_data(plt_models_df)

mltdCI_models_df <- melt(pltCI_models_df, id.vars="id")
for (row_ix in 1:nrow(mltdCI_models_df)) {
    for (type in c("Upper", "Lower")) {
        if (length(var_components <- unlist(strsplit(
                as.character(mltdCI_models_df[row_ix, "variable"]), type))) > 1) {
            #print(sprintf("row_ix:%d; type:%s; ", row_ix, type))
            mltdCI_models_df[row_ix, "label"] <- var_components[1]
            mltdCI_models_df[row_ix, "data"] <- 
                unlist(strsplit(var_components[2], "[.]"))[2]
            mltdCI_models_df[row_ix, "type"] <- type
            break
        }
    }    
}
wideCI_models_df <- reshape(subset(mltdCI_models_df, select=-variable), 
                            timevar="type", 
        idvar=setdiff(names(mltdCI_models_df), c("type", "value", "variable")), 
                            direction="wide")
#print(wideCI_models_df)
mrgdCI_models_df <- merge(wideCI_models_df, mltd_models_df, all.x=TRUE)
#print(mrgdCI_models_df)

# Merge stats back in if CIs don't exist
goback_vars <- c()
for (var in unique(mltd_models_df$label)) {
    for (type in unique(mltd_models_df$data)) {
        var_type <- paste0(var, ".", type)
        # if this data is already present, next
        if (var_type %in% unique(paste(mltd_models_df$label, mltd_models_df$data,
                                       sep=".")))
            next
        #print(sprintf("var_type:%s", var_type))
        goback_vars <- c(goback_vars, var_type)
    }
}

if (length(goback_vars) > 0) {
    mltd_goback_df <- build_statsCI_data(glb_models_df[, c("id", goback_vars)])
    mltd_models_df <- rbind(mltd_models_df, mltd_goback_df)
}

# mltd_models_df <- merge(mltd_models_df, glb_models_df[, c("id", "model_method")], 
#                         all.x=TRUE)

png(paste0(glbOut$pfx, "models_bar.png"), width=480*3, height=480*2)
#print(gp <- myplot_bar(mltd_models_df, "id", "value", colorcol_name="model_method") + 
print(gp <- myplot_bar(df=mltd_models_df, xcol_name="id", ycol_names="value") + 
        geom_errorbar(data=mrgdCI_models_df, 
            mapping=aes(x=mdl_id, ymax=value.Upper, ymin=value.Lower), width=0.5) + 
          facet_grid(label ~ data, scales="free") + 
          theme(axis.text.x = element_text(angle = 90,vjust = 0.5)))

## Warning: Removed 4 rows containing missing values (geom_errorbar).

dev.off()

## quartz_off_screen 
##                 2

print(gp)

## Warning: Removed 4 rows containing missing values (geom_errorbar).

dsp_models_cols <- c("id", 
                    glbMdlMetricsEval[glbMdlMetricsEval %in% names(glb_models_df)],
                    grep("opt.", names(glb_models_df), fixed = TRUE, value = TRUE)) 
# if (glb_is_classification && glb_is_binomial) 
#     dsp_models_cols <- c(dsp_models_cols, "opt.prob.threshold.OOB")
print(dsp_models_df <- orderBy(get_model_sel_frmla(), glb_models_df)[, dsp_models_cols])

##                                id max.Accuracy.OOB max.AUCROCR.OOB
## 5 Interact.High.cor.Y##rcv#glmnet        0.5732143       0.3571392
## 8                  All.X##rcv#glm        0.5723214       0.3358234
## 4            Max.cor.Y##rcv#rpart        0.5633929       0.3774772
## 3      Max.cor.Y.rcv.1X1###glmnet        0.5633929       0.3658672
## 6           Low.cor.X##rcv#glmnet        0.5544643       0.3184875
## 7               All.X##rcv#glmnet        0.5544643       0.3184875
## 2       Random###myrandom_classfr        0.4696429       0.5191202
## 1             MFO###myMFO_classfr        0.4696429       0.5000000
##   max.AUCpROC.OOB max.Accuracy.fit opt.prob.threshold.fit
## 5       0.6031353        0.6058167                   0.65
## 8       0.5947690        0.6029697                   0.65
## 4       0.5896897        0.6000450                   0.55
## 3       0.5896897        0.5721673                   0.60
## 6       0.6237374        0.6241754                   0.60
## 7       0.6237374        0.6241754                   0.60
## 2       0.5235690        0.4700989                   0.55
## 1       0.5000000        0.4700989                   0.50
##   opt.prob.threshold.OOB
## 5                   0.65
## 8                   0.75
## 4                   0.55
## 3                   0.60
## 6                   0.70
## 7                   0.70
## 2                   0.55
## 1                   0.50

# print(myplot_radar(radar_inp_df = dsp_models_df))
print("Metrics used for model selection:"); print(get_model_sel_frmla())

## [1] "Metrics used for model selection:"

## ~-max.Accuracy.OOB - max.AUCROCR.OOB - max.AUCpROC.OOB - max.Accuracy.fit - 
##     opt.prob.threshold.OOB
## <environment: 0x7fd98e2d5a30>

print(sprintf("Best model id: %s", dsp_models_df[1, "id"]))

## [1] "Best model id: Interact.High.cor.Y##rcv#glmnet"

glb_get_predictions <- function(df, mdl_id, rsp_var, prob_threshold_def=NULL, verbose=FALSE) {
    mdl <- glb_models_lst[[mdl_id]]
    
    clmnNames <- mygetPredictIds(rsp_var, mdl_id)
    predct_var_name <- clmnNames$value        
    predct_prob_var_name <- clmnNames$prob
    predct_accurate_var_name <- clmnNames$is.acc
    predct_error_var_name <- clmnNames$err
    predct_erabs_var_name <- clmnNames$err.abs

    if (glb_is_regression) {
        df[, predct_var_name] <- predict(mdl, newdata=df, type="raw")
        if (verbose) print(myplot_scatter(df, glb_rsp_var, predct_var_name) + 
                  facet_wrap(reformulate(glbFeatsCategory), scales = "free") + 
                  stat_smooth(method="glm"))

        df[, predct_error_var_name] <- df[, predct_var_name] - df[, glb_rsp_var]
        if (verbose) print(myplot_scatter(df, predct_var_name, predct_error_var_name) + 
                  #facet_wrap(reformulate(glbFeatsCategory), scales = "free") + 
                  stat_smooth(method="auto"))
        if (verbose) print(myplot_scatter(df, glb_rsp_var, predct_error_var_name) + 
                  #facet_wrap(reformulate(glbFeatsCategory), scales = "free") + 
                  stat_smooth(method="glm"))
        
        df[, predct_erabs_var_name] <- abs(df[, predct_error_var_name])
        if (verbose) print(head(orderBy(reformulate(c("-", predct_erabs_var_name)), df)))
        
        df[, predct_accurate_var_name] <- (df[, glb_rsp_var] == df[, predct_var_name])
    }

    if (glb_is_classification && glb_is_binomial) {
        prob_threshold <- glb_models_df[glb_models_df$id == mdl_id, 
                                        "opt.prob.threshold.OOB"]
        if (is.null(prob_threshold) || is.na(prob_threshold)) {
            warning("Using default probability threshold: ", prob_threshold_def)
            if (is.null(prob_threshold <- prob_threshold_def))
                stop("Default probability threshold is NULL")
        }
        
        df[, predct_prob_var_name] <- predict(mdl, newdata = df, type = "prob")[, 2]
        df[, predct_var_name] <- 
                factor(levels(df[, glb_rsp_var])[
                    (df[, predct_prob_var_name] >=
                        prob_threshold) * 1 + 1], levels(df[, glb_rsp_var]))
    
#         if (verbose) print(myplot_scatter(df, glb_rsp_var, predct_var_name) + 
#                   facet_wrap(reformulate(glbFeatsCategory), scales = "free") + 
#                   stat_smooth(method="glm"))

        df[, predct_error_var_name] <- df[, predct_var_name] != df[, glb_rsp_var]
#         if (verbose) print(myplot_scatter(df, predct_var_name, predct_error_var_name) + 
#                   #facet_wrap(reformulate(glbFeatsCategory), scales = "free") + 
#                   stat_smooth(method="auto"))
#         if (verbose) print(myplot_scatter(df, glb_rsp_var, predct_error_var_name) + 
#                   #facet_wrap(reformulate(glbFeatsCategory), scales = "free") + 
#                   stat_smooth(method="glm"))
        
        # if prediction is a TP (true +ve), measure distance from 1.0
        tp <- which((df[, predct_var_name] == df[, glb_rsp_var]) &
                    (df[, predct_var_name] == levels(df[, glb_rsp_var])[2]))
        df[tp, predct_erabs_var_name] <- abs(1 - df[tp, predct_prob_var_name])
        #rowIx <- which.max(df[tp, predct_erabs_var_name]); df[tp, c(glbFeatsId, glb_rsp_var, predct_var_name, predct_prob_var_name, predct_erabs_var_name)][rowIx, ]
        
        # if prediction is a TN (true -ve), measure distance from 0.0
        tn <- which((df[, predct_var_name] == df[, glb_rsp_var]) &
                    (df[, predct_var_name] == levels(df[, glb_rsp_var])[1]))
        df[tn, predct_erabs_var_name] <- abs(0 - df[tn, predct_prob_var_name])
        #rowIx <- which.max(df[tn, predct_erabs_var_name]); df[tn, c(glbFeatsId, glb_rsp_var, predct_var_name, predct_prob_var_name, predct_erabs_var_name)][rowIx, ]
        
        # if prediction is a FP (flse +ve), measure distance from 0.0
        fp <- which((df[, predct_var_name] != df[, glb_rsp_var]) &
                    (df[, predct_var_name] == levels(df[, glb_rsp_var])[2]))
        df[fp, predct_erabs_var_name] <- abs(0 - df[fp, predct_prob_var_name])
        #rowIx <- which.max(df[fp, predct_erabs_var_name]); df[fp, c(glbFeatsId, glb_rsp_var, predct_var_name, predct_prob_var_name, predct_erabs_var_name)][rowIx, ]
        
        # if prediction is a FN (flse -ve), measure distance from 1.0
        fn <- which((df[, predct_var_name] != df[, glb_rsp_var]) &
                    (df[, predct_var_name] == levels(df[, glb_rsp_var])[1]))
        df[fn, predct_erabs_var_name] <- abs(1 - df[fn, predct_prob_var_name])
        #rowIx <- which.max(df[fn, predct_erabs_var_name]); df[fn, c(glbFeatsId, glb_rsp_var, predct_var_name, predct_prob_var_name, predct_erabs_var_name)][rowIx, ]

        
        if (verbose) print(head(orderBy(reformulate(c("-", predct_erabs_var_name)), df)))
        
        df[, predct_accurate_var_name] <- (df[, glb_rsp_var] == df[, predct_var_name])
    }    
    
    if (glb_is_classification && !glb_is_binomial) {
        df[, predct_var_name] <- predict(mdl, newdata = df, type = "raw")
        probCls <- predict(mdl, newdata = df, type = "prob")        
        df[, predct_prob_var_name] <- NA
        for (cls in names(probCls)) {
            mask <- (df[, predct_var_name] == cls)
            df[mask, predct_prob_var_name] <- probCls[mask, cls]
        }    
        if (verbose) print(myplot_histogram(df, predct_prob_var_name, 
                                            fill_col_name = predct_var_name))
        if (verbose) print(myplot_histogram(df, predct_prob_var_name, 
                                            facet_frmla = paste0("~", glb_rsp_var)))
        
        df[, predct_error_var_name] <- df[, predct_var_name] != df[, glb_rsp_var]
        
        # if prediction is erroneous, measure predicted class prob from actual class prob
        df[, predct_erabs_var_name] <- 0
        for (cls in names(probCls)) {
            mask <- (df[, glb_rsp_var] == cls) & (df[, predct_error_var_name])
            df[mask, predct_erabs_var_name] <- probCls[mask, cls]
        }    

        df[, predct_accurate_var_name] <- (df[, glb_rsp_var] == df[, predct_var_name])        
    }

    return(df)
}    

#stop(here"); glb2Sav(); glbObsAll <- savObsAll; glbObsTrn <- savObsTrn; glbObsFit <- savObsFit; glbObsOOB <- savObsOOB; sav_models_df <- glb_models_df; glb_models_df <- sav_models_df; glb_featsimp_df <- sav_featsimp_df    

myget_category_stats <- function(obs_df, mdl_id, label) {
    require(dplyr)
    require(lazyeval)
    
    predct_var_name <- mygetPredictIds(glb_rsp_var, mdl_id)$value        
    predct_error_var_name <- mygetPredictIds(glb_rsp_var, mdl_id)$err.abs
    
    if (!predct_var_name %in% names(obs_df))
        obs_df <- glb_get_predictions(obs_df, mdl_id, glb_rsp_var)
    
    tmp_obs_df <- obs_df[, c(glbFeatsCategory, glb_rsp_var, 
                             predct_var_name, predct_error_var_name)]
#     tmp_obs_df <- obs_df %>%
#         dplyr::select_(glbFeatsCategory, glb_rsp_var, predct_var_name, predct_error_var_name) 
    #dplyr::rename(startprice.log10.predict.RFE.X.glmnet.err=error_abs_OOB)
    names(tmp_obs_df)[length(names(tmp_obs_df))] <- paste0("err.abs.", label)
    
    ret_ctgry_df <- tmp_obs_df %>%
        dplyr::group_by_(glbFeatsCategory) %>%
        dplyr::summarise_(#interp(~sum(abs(var)), var=as.name(glb_rsp_var)), 
            interp(~sum(var), var=as.name(paste0("err.abs.", label))), 
            interp(~mean(var), var=as.name(paste0("err.abs.", label))),
            interp(~n()))
    names(ret_ctgry_df) <- c(glbFeatsCategory, 
                             #paste0(glb_rsp_var, ".abs.", label, ".sum"),
                             paste0("err.abs.", label, ".sum"),                             
                             paste0("err.abs.", label, ".mean"), 
                             paste0(".n.", label))
    ret_ctgry_df <- dplyr::ungroup(ret_ctgry_df)
    #colSums(ret_ctgry_df[, -grep(glbFeatsCategory, names(ret_ctgry_df))])
    
    return(ret_ctgry_df)    
}
#print(colSums((ctgry_df <- myget_category_stats(obs_df=glbObsFit, mdl_id="", label="fit"))[, -grep(glbFeatsCategory, names(ctgry_df))]))

if (!is.null(glb_mdl_ensemble)) {
    fit.models_2_chunk_df <- myadd_chunk(fit.models_2_chunk_df, 
                            paste0("fit.models_2_", mdl_id_pfx), major.inc = TRUE, 
                                                label.minor = "ensemble")
    
    mdl_id_pfx <- "Ensemble"

    if (#(glb_is_regression) | 
        ((glb_is_classification) & (!glb_is_binomial)))
        stop("Ensemble models not implemented yet for multinomial classification")
    
    mygetEnsembleAutoMdlIds <- function() {
        tmp_models_df <- orderBy(get_model_sel_frmla(), glb_models_df)
        row.names(tmp_models_df) <- tmp_models_df$id
        mdl_threshold_pos <- 
            min(which(grepl("MFO|Random|Baseline", tmp_models_df$id))) - 1
        mdlIds <- tmp_models_df$id[1:mdl_threshold_pos]
        return(mdlIds[!grepl("Ensemble", mdlIds)])
    }
    
    if (glb_mdl_ensemble == "auto") {
        glb_mdl_ensemble <- mygetEnsembleAutoMdlIds()
        mdl_id_pfx <- paste0(mdl_id_pfx, ".auto")        
    } else if (grepl("^%<d-%", glb_mdl_ensemble)) {
        glb_mdl_ensemble <- eval(parse(text =
                        str_trim(unlist(strsplit(glb_mdl_ensemble, "%<d-%"))[2])))
    }
    
    for (mdl_id in glb_mdl_ensemble) {
        if (!(mdl_id %in% names(glb_models_lst))) {
            warning("Model ", mdl_id, " in glb_model_ensemble not found !")
            next
        }
        glbObsFit <- glb_get_predictions(df = glbObsFit, mdl_id, glb_rsp_var)
        glbObsOOB <- glb_get_predictions(df = glbObsOOB, mdl_id, glb_rsp_var)
    }
    
#mdl_id_pfx <- "Ensemble.RFE"; mdlId <- paste0(mdl_id_pfx, ".glmnet")
#glb_mdl_ensemble <- gsub(mygetPredictIds$value, "", grep("RFE\\.X\\.(?!Interact)", row.names(glb_featsimp_df), perl = TRUE, value = TRUE), fixed = TRUE)
#varImp(glb_models_lst[[mdlId]])
    
#cor_df <- data.frame(cor=cor(glbObsFit[, glb_rsp_var], glbObsFit[, paste(mygetPredictIds$value, glb_mdl_ensemble)], use="pairwise.complete.obs"))
#glbObsFit <- glb_get_predictions(df=glbObsFit, "Ensemble.glmnet", glb_rsp_var);print(colSums((ctgry_df <- myget_category_stats(obs_df=glbObsFit, mdl_id="Ensemble.glmnet", label="fit"))[, -grep(glbFeatsCategory, names(ctgry_df))]))
    
    ### bid0_sp
    #  Better than MFO; models.n=28; min.RMSE.fit=0.0521233; err.abs.fit.sum=7.3631895
    #  old: Top x from auto; models.n= 5; min.RMSE.fit=0.06311047; err.abs.fit.sum=9.5937080
    #  RFE only ;       models.n=16; min.RMSE.fit=0.05148588; err.abs.fit.sum=7.2875091
    #  RFE subset only ;models.n= 5; min.RMSE.fit=0.06040702; err.abs.fit.sum=9.059088
    #  RFE subset only ;models.n= 9; min.RMSE.fit=0.05933167; err.abs.fit.sum=8.7421288
    #  RFE subset only ;models.n=15; min.RMSE.fit=0.0584607; err.abs.fit.sum=8.5902066
    #  RFE subset only ;models.n=17; min.RMSE.fit=0.05496899; err.abs.fit.sum=8.0170431
    #  RFE subset only ;models.n=18; min.RMSE.fit=0.05441577; err.abs.fit.sum=7.837223
    #  RFE subset only ;models.n=16; min.RMSE.fit=0.05441577; err.abs.fit.sum=7.837223
    ### bid0_sp
    ### bid1_sp
    # "auto"; err.abs.fit.sum=76.699774; min.RMSE.fit=0.2186429
    # "RFE.X.*"; err.abs.fit.sum=; min.RMSE.fit=0.221114
    ### bid1_sp

    indepVar <- paste(mygetPredictIds(glb_rsp_var)$value, glb_mdl_ensemble, sep = "")
    if (glb_is_classification)
        indepVar <- paste(indepVar, ".prob", sep = "")
    # Some models in glb_mdl_ensemble might not be fitted e.g. RFE.X.Interact
    indepVar <- intersect(indepVar, names(glbObsFit))
    
#     indepVar <- grep(mygetPredictIds(glb_rsp_var)$value, names(glbObsFit), fixed=TRUE, value=TRUE)
#     if (glb_is_regression)
#         indepVar <- indepVar[!grepl("(err\\.abs|accurate)$", indepVar)]
#     if (glb_is_classification && glb_is_binomial)
#         indepVar <- grep("prob$", indepVar, value=TRUE) else
#         indepVar <- indepVar[!grepl("err$", indepVar)]

    #rfe_fit_ens_results <- myrun_rfe(glbObsFit, indepVar)
    
    for (method in c("glm", "glmnet")) {
        for (trainControlMethod in 
             c("boot", "boot632", "cv", "repeatedcv"
               #, "LOOCV" # tuneLength * nrow(fitDF)
               , "LGOCV", "adaptive_cv"
               #, "adaptive_boot"  #error: adaptive$min should be less than 3 
               #, "adaptive_LGOCV" #error: adaptive$min should be less than 3 
               )) {
            #sav_models_df <- glb_models_df; all.equal(sav_models_df, glb_models_df)
            #glb_models_df <- sav_models_df; print(glb_models_df$id)
                
            if ((method == "glm") && (trainControlMethod != "repeatedcv"))
                # glm used only to identify outliers
                next
            
            ret_lst <- myfit_mdl(
                mdl_specs_lst = myinit_mdl_specs_lst(mdl_specs_lst = list(
                    id.prefix = paste0(mdl_id_pfx, ".", trainControlMethod), 
                    type = glb_model_type, tune.df = NULL,
                    trainControl.method = trainControlMethod,
                    trainControl.number = glb_rcv_n_folds,
                    trainControl.repeats = glb_rcv_n_repeats,
                    trainControl.classProbs = glb_is_classification,
                    trainControl.summaryFunction = glbMdlMetricSummaryFn,
                    train.metric = glbMdlMetricSummary, 
                    train.maximize = glbMdlMetricMaximize,    
                    train.method = method)),
                indepVar = indepVar, rsp_var = glb_rsp_var, 
                fit_df = glbObsFit, OOB_df = glbObsOOB)
        }
    }
    dsp_models_df <- get_dsp_models_df()
}

if (is.null(glbMdlSelId)) 
    glbMdlSelId <- dsp_models_df[1, "id"] else 
    print(sprintf("User specified selection: %s", glbMdlSelId))

## [1] "User specified selection: All.X##rcv#glmnet"

myprint_mdl(glb_sel_mdl <- glb_models_lst[[glbMdlSelId]])

##             Length Class      Mode     
## a0             78  -none-     numeric  
## beta        19734  dgCMatrix  S4       
## df             78  -none-     numeric  
## dim             2  -none-     numeric  
## lambda         78  -none-     numeric  
## dev.ratio      78  -none-     numeric  
## nulldev         1  -none-     numeric  
## npasses         1  -none-     numeric  
## jerr            1  -none-     numeric  
## offset          1  -none-     logical  
## classnames      2  -none-     character
## call            5  -none-     call     
## nobs            1  -none-     numeric  
## lambdaOpt       1  -none-     numeric  
## xNames        253  -none-     character
## problemType     1  -none-     character
## tuneValue       2  data.frame list     
## obsLevels       2  -none-     character
## [1] "min lambda > lambdaOpt:"
##       (Intercept)      Gender.fctrM     Hhold.fctrMKy     Hhold.fctrPKn 
##      0.1759341202     -0.0577396529     -0.0655451865      0.3282606962 
##   Q101163.fctrDad   Q101163.fctrMom    Q109244.fctrNo   Q109244.fctrYes 
##     -0.0462347606      0.0361283260     -0.3170070971      0.7976156342 
##    Q110740.fctrPC    Q113181.fctrNo   Q113181.fctrYes    Q115611.fctrNo 
##     -0.0120813108      0.0348921948     -0.0798654082      0.0770473552 
##   Q115611.fctrYes Q116881.fctrHappy Q116881.fctrRight    Q118232.fctrId 
##     -0.3105852699      0.0087382029     -0.0814535814      0.0180660706 
##    Q119851.fctrNo    Q120379.fctrNo   Q120379.fctrYes     Q98197.fctrNo 
##     -0.0297897358     -0.0006993182      0.0272478036      0.1639383807 
##    Q98197.fctrYes     Q98869.fctrNo     Q99480.fctrNo 
##     -0.0341915807      0.1609634790      0.0576623212 
## [1] "max lambda < lambdaOpt:"
##       (Intercept)      Gender.fctrM     Hhold.fctrMKy     Hhold.fctrPKn 
##       0.180590790      -0.063356510      -0.071963008       0.358397790 
##     Income.fctr.Q   Q101163.fctrDad   Q101163.fctrMom    Q109244.fctrNo 
##      -0.003003778      -0.055383160       0.041352778      -0.326547933 
##   Q109244.fctrYes    Q110740.fctrPC    Q113181.fctrNo   Q113181.fctrYes 
##       0.811987955      -0.023695743       0.041178168      -0.082431818 
##    Q115611.fctrNo   Q115611.fctrYes    Q115899.fctrCs Q116881.fctrHappy 
##       0.080608650      -0.318215302       0.001114558       0.019171363 
## Q116881.fctrRight    Q118232.fctrId    Q119851.fctrNo    Q120379.fctrNo 
##      -0.090653797       0.032794176      -0.043521175      -0.004314002 
##   Q120379.fctrYes    Q121699.fctrNo    Q122771.fctrPt     Q98197.fctrNo 
##       0.038379682      -0.010439127      -0.000907088       0.171596860 
##    Q98197.fctrYes     Q98869.fctrNo     Q99480.fctrNo 
##      -0.034226771       0.174722108       0.068918234

## [1] TRUE

# From here to save(), this should all be in one function
#   these are executed in the same seq twice more:
#       fit.data.training & predict.data.new chunks
print(sprintf("%s fit prediction diagnostics:", glbMdlSelId))

## [1] "All.X##rcv#glmnet fit prediction diagnostics:"

glbObsFit <- glb_get_predictions(df = glbObsFit, mdl_id = glbMdlSelId, 
                                 rsp_var = glb_rsp_var)
print(sprintf("%s OOB prediction diagnostics:", glbMdlSelId))

## [1] "All.X##rcv#glmnet OOB prediction diagnostics:"

glbObsOOB <- glb_get_predictions(df = glbObsOOB, mdl_id = glbMdlSelId, 
                                     rsp_var = glb_rsp_var)

print(glb_featsimp_df <- myget_feats_importance(mdl = glb_sel_mdl, featsimp_df = NULL))

##                                All.X..rcv.glmnet.imp          imp
## Q109244.fctrYes                         100.00000000 100.00000000
## Hhold.fctrPKn                            41.56642724  41.56642724
## Q109244.fctrNo                           39.80933610  39.80933610
## Q115611.fctrYes                          38.97373582  38.97373582
## Q98197.fctrNo                            20.63341630  20.63341630
## Q98869.fctrNo                            20.36490400  20.36490400
## Q116881.fctrRight                        10.34339619  10.34339619
## Q113181.fctrYes                          10.03215580  10.03215580
## Q115611.fctrNo                            9.69659664   9.69659664
## Hhold.fctrMKy                             8.30653632   8.30653632
## Q99480.fctrNo                             7.40277198   7.40277198
## Gender.fctrM                              7.31671883   7.31671883
## Q101163.fctrDad                           5.93777650   5.93777650
## Q101163.fctrMom                           4.60717672   4.60717672
## Q113181.fctrNo                            4.47059621   4.47059621
## Q98197.fctrYes                            4.27686282   4.27686282
## Q119851.fctrNo                            3.95883221   3.95883221
## Q120379.fctrYes                           3.59678995   3.59678995
## Q118232.fctrId                            2.50949804   2.50949804
## Q110740.fctrPC                            1.70814173   1.70814173
## Q116881.fctrHappy                         1.26997411   1.26997411
## Q121699.fctrNo                            0.17720767   0.17720767
## Q120379.fctrNo                            0.14882265   0.14882265
## Income.fctr.Q                             0.05099013   0.05099013
## Q115899.fctrCs                            0.01891999   0.01891999
## Q122771.fctrPt                            0.01539812   0.01539812
## .rnorm                                    0.00000000   0.00000000
## Edn.fctr.L                                0.00000000   0.00000000
## Edn.fctr.Q                                0.00000000   0.00000000
## Edn.fctr.C                                0.00000000   0.00000000
## Edn.fctr^4                                0.00000000   0.00000000
## Edn.fctr^5                                0.00000000   0.00000000
## Edn.fctr^6                                0.00000000   0.00000000
## Edn.fctr^7                                0.00000000   0.00000000
## Gender.fctrF                              0.00000000   0.00000000
## Hhold.fctrMKn                             0.00000000   0.00000000
## Hhold.fctrPKy                             0.00000000   0.00000000
## Hhold.fctrSKn                             0.00000000   0.00000000
## Hhold.fctrSKy                             0.00000000   0.00000000
## Income.fctr.L                             0.00000000   0.00000000
## Income.fctr.C                             0.00000000   0.00000000
## Income.fctr^4                             0.00000000   0.00000000
## Income.fctr^5                             0.00000000   0.00000000
## Income.fctr^6                             0.00000000   0.00000000
## Q100010.fctrNo                            0.00000000   0.00000000
## Q100010.fctrYes                           0.00000000   0.00000000
## Q100562.fctrNo                            0.00000000   0.00000000
## Q100562.fctrYes                           0.00000000   0.00000000
## Q100680.fctrNo                            0.00000000   0.00000000
## Q100680.fctrYes                           0.00000000   0.00000000
## Q100689.fctrNo                            0.00000000   0.00000000
## Q100689.fctrYes                           0.00000000   0.00000000
## Q101162.fctrOptimist                      0.00000000   0.00000000
## Q101162.fctrPessimist                     0.00000000   0.00000000
## Q101596.fctrNo                            0.00000000   0.00000000
## Q101596.fctrYes                           0.00000000   0.00000000
## Q102089.fctrOwn                           0.00000000   0.00000000
## Q102089.fctrRent                          0.00000000   0.00000000
## Q102289.fctrNo                            0.00000000   0.00000000
## Q102289.fctrYes                           0.00000000   0.00000000
## Q102674.fctrNo                            0.00000000   0.00000000
## Q102674.fctrYes                           0.00000000   0.00000000
## Q102687.fctrNo                            0.00000000   0.00000000
## Q102687.fctrYes                           0.00000000   0.00000000
## Q102906.fctrNo                            0.00000000   0.00000000
## Q102906.fctrYes                           0.00000000   0.00000000
## Q103293.fctrNo                            0.00000000   0.00000000
## Q103293.fctrYes                           0.00000000   0.00000000
## Q104996.fctrNo                            0.00000000   0.00000000
## Q104996.fctrYes                           0.00000000   0.00000000
## Q105655.fctrNo                            0.00000000   0.00000000
## Q105655.fctrYes                           0.00000000   0.00000000
## Q105840.fctrNo                            0.00000000   0.00000000
## Q105840.fctrYes                           0.00000000   0.00000000
## Q106042.fctrNo                            0.00000000   0.00000000
## Q106042.fctrYes                           0.00000000   0.00000000
## Q106272.fctrNo                            0.00000000   0.00000000
## Q106272.fctrYes                           0.00000000   0.00000000
## Q106388.fctrNo                            0.00000000   0.00000000
## Q106388.fctrYes                           0.00000000   0.00000000
## Q106389.fctrNo                            0.00000000   0.00000000
## Q106389.fctrYes                           0.00000000   0.00000000
## Q106993.fctrNo                            0.00000000   0.00000000
## Q106993.fctrYes                           0.00000000   0.00000000
## Q106997.fctrGr                            0.00000000   0.00000000
## Q106997.fctrYy                            0.00000000   0.00000000
## Q107491.fctrNo                            0.00000000   0.00000000
## Q107491.fctrYes                           0.00000000   0.00000000
## Q107869.fctrNo                            0.00000000   0.00000000
## Q107869.fctrYes                           0.00000000   0.00000000
## Q108342.fctrIn-person                     0.00000000   0.00000000
## Q108342.fctrOnline                        0.00000000   0.00000000
## Q108343.fctrNo                            0.00000000   0.00000000
## Q108343.fctrYes                           0.00000000   0.00000000
## Q108617.fctrNo                            0.00000000   0.00000000
## Q108617.fctrYes                           0.00000000   0.00000000
## Q108754.fctrNo                            0.00000000   0.00000000
## Q108754.fctrYes                           0.00000000   0.00000000
## Q108855.fctrUmm...                        0.00000000   0.00000000
## Q108855.fctrYes!                          0.00000000   0.00000000
## Q108856.fctrSocialize                     0.00000000   0.00000000
## Q108856.fctrSpace                         0.00000000   0.00000000
## Q108950.fctrCautious                      0.00000000   0.00000000
## Q108950.fctrRisk-friendly                 0.00000000   0.00000000
## Q109367.fctrNo                            0.00000000   0.00000000
## Q109367.fctrYes                           0.00000000   0.00000000
## Q110740.fctrMac                           0.00000000   0.00000000
## Q111220.fctrNo                            0.00000000   0.00000000
## Q111220.fctrYes                           0.00000000   0.00000000
## Q111580.fctrDemanding                     0.00000000   0.00000000
## Q111580.fctrSupportive                    0.00000000   0.00000000
## Q111848.fctrNo                            0.00000000   0.00000000
## Q111848.fctrYes                           0.00000000   0.00000000
## Q112270.fctrNo                            0.00000000   0.00000000
## Q112270.fctrYes                           0.00000000   0.00000000
## Q112478.fctrNo                            0.00000000   0.00000000
## Q112478.fctrYes                           0.00000000   0.00000000
## Q112512.fctrNo                            0.00000000   0.00000000
## Q112512.fctrYes                           0.00000000   0.00000000
## Q113583.fctrTalk                          0.00000000   0.00000000
## Q113583.fctrTunes                         0.00000000   0.00000000
## Q113584.fctrPeople                        0.00000000   0.00000000
## Q113584.fctrTechnology                    0.00000000   0.00000000
## Q113992.fctrNo                            0.00000000   0.00000000
## Q113992.fctrYes                           0.00000000   0.00000000
## Q114152.fctrNo                            0.00000000   0.00000000
## Q114152.fctrYes                           0.00000000   0.00000000
## Q114386.fctrMysterious                    0.00000000   0.00000000
## Q114386.fctrTMI                           0.00000000   0.00000000
## Q114517.fctrNo                            0.00000000   0.00000000
## Q114517.fctrYes                           0.00000000   0.00000000
## Q114748.fctrNo                            0.00000000   0.00000000
## Q114748.fctrYes                           0.00000000   0.00000000
## Q114961.fctrNo                            0.00000000   0.00000000
## Q114961.fctrYes                           0.00000000   0.00000000
## Q115195.fctrNo                            0.00000000   0.00000000
## Q115195.fctrYes                           0.00000000   0.00000000
## Q115390.fctrNo                            0.00000000   0.00000000
## Q115390.fctrYes                           0.00000000   0.00000000
## Q115602.fctrNo                            0.00000000   0.00000000
## Q115602.fctrYes                           0.00000000   0.00000000
## Q115610.fctrNo                            0.00000000   0.00000000
## Q115610.fctrYes                           0.00000000   0.00000000
## Q115777.fctrEnd                           0.00000000   0.00000000
## Q115777.fctrStart                         0.00000000   0.00000000
## Q115899.fctrMe                            0.00000000   0.00000000
## Q116197.fctrA.M.                          0.00000000   0.00000000
## Q116197.fctrP.M.                          0.00000000   0.00000000
## Q116441.fctrNo                            0.00000000   0.00000000
## Q116441.fctrYes                           0.00000000   0.00000000
## Q116448.fctrNo                            0.00000000   0.00000000
## Q116448.fctrYes                           0.00000000   0.00000000
## Q116601.fctrNo                            0.00000000   0.00000000
## Q116601.fctrYes                           0.00000000   0.00000000
## Q116797.fctrNo                            0.00000000   0.00000000
## Q116797.fctrYes                           0.00000000   0.00000000
## Q116953.fctrNo                            0.00000000   0.00000000
## Q116953.fctrYes                           0.00000000   0.00000000
## Q117186.fctrCool headed                   0.00000000   0.00000000
## Q117186.fctrHot headed                    0.00000000   0.00000000
## Q117193.fctrOdd hours                     0.00000000   0.00000000
## Q117193.fctrStandard hours                0.00000000   0.00000000
## Q118117.fctrNo                            0.00000000   0.00000000
## Q118117.fctrYes                           0.00000000   0.00000000
## Q118232.fctrPr                            0.00000000   0.00000000
## Q118233.fctrNo                            0.00000000   0.00000000
## Q118233.fctrYes                           0.00000000   0.00000000
## Q118237.fctrNo                            0.00000000   0.00000000
## Q118237.fctrYes                           0.00000000   0.00000000
## Q118892.fctrNo                            0.00000000   0.00000000
## Q118892.fctrYes                           0.00000000   0.00000000
## Q119334.fctrNo                            0.00000000   0.00000000
## Q119334.fctrYes                           0.00000000   0.00000000
## Q119650.fctrGiving                        0.00000000   0.00000000
## Q119650.fctrReceiving                     0.00000000   0.00000000
## Q119851.fctrYes                           0.00000000   0.00000000
## Q120012.fctrNo                            0.00000000   0.00000000
## Q120012.fctrYes                           0.00000000   0.00000000
## Q120014.fctrNo                            0.00000000   0.00000000
## Q120014.fctrYes                           0.00000000   0.00000000
## Q120194.fctrStudy first                   0.00000000   0.00000000
## Q120194.fctrTry first                     0.00000000   0.00000000
## Q120472.fctrArt                           0.00000000   0.00000000
## Q120472.fctrScience                       0.00000000   0.00000000
## Q120650.fctrNo                            0.00000000   0.00000000
## Q120650.fctrYes                           0.00000000   0.00000000
## Q120978.fctrNo                            0.00000000   0.00000000
## Q120978.fctrYes                           0.00000000   0.00000000
## Q121011.fctrNo                            0.00000000   0.00000000
## Q121011.fctrYes                           0.00000000   0.00000000
## Q121699.fctrYes                           0.00000000   0.00000000
## Q121700.fctrNo                            0.00000000   0.00000000
## Q121700.fctrYes                           0.00000000   0.00000000
## Q122120.fctrNo                            0.00000000   0.00000000
## Q122120.fctrYes                           0.00000000   0.00000000
## Q122769.fctrNo                            0.00000000   0.00000000
## Q122769.fctrYes                           0.00000000   0.00000000
## Q122770.fctrNo                            0.00000000   0.00000000
## Q122770.fctrYes                           0.00000000   0.00000000
## Q122771.fctrPc                            0.00000000   0.00000000
## Q123464.fctrNo                            0.00000000   0.00000000
## Q123464.fctrYes                           0.00000000   0.00000000
## Q123621.fctrNo                            0.00000000   0.00000000
## Q123621.fctrYes                           0.00000000   0.00000000
## Q124122.fctrNo                            0.00000000   0.00000000
## Q124122.fctrYes                           0.00000000   0.00000000
## Q124742.fctrNo                            0.00000000   0.00000000
## Q124742.fctrYes                           0.00000000   0.00000000
## Q96024.fctrNo                             0.00000000   0.00000000
## Q96024.fctrYes                            0.00000000   0.00000000
## Q98059.fctrOnly-child                     0.00000000   0.00000000
## Q98059.fctrYes                            0.00000000   0.00000000
## Q98078.fctrNo                             0.00000000   0.00000000
## Q98078.fctrYes                            0.00000000   0.00000000
## Q98578.fctrNo                             0.00000000   0.00000000
## Q98578.fctrYes                            0.00000000   0.00000000
## Q98869.fctrYes                            0.00000000   0.00000000
## Q99480.fctrYes                            0.00000000   0.00000000
## Q99581.fctrNo                             0.00000000   0.00000000
## Q99581.fctrYes                            0.00000000   0.00000000
## Q99716.fctrNo                             0.00000000   0.00000000
## Q99716.fctrYes                            0.00000000   0.00000000
## Q99982.fctrCheck!                         0.00000000   0.00000000
## Q99982.fctrNope                           0.00000000   0.00000000
## YOB.Age.fctr.L                            0.00000000   0.00000000
## YOB.Age.fctr.Q                            0.00000000   0.00000000
## YOB.Age.fctr.C                            0.00000000   0.00000000
## YOB.Age.fctr^4                            0.00000000   0.00000000
## YOB.Age.fctr^5                            0.00000000   0.00000000
## YOB.Age.fctr^6                            0.00000000   0.00000000
## YOB.Age.fctr^7                            0.00000000   0.00000000
## YOB.Age.fctr^8                            0.00000000   0.00000000
## Hhold.fctrN:.clusterid.fctr2              0.00000000   0.00000000
## Hhold.fctrMKn:.clusterid.fctr2            0.00000000   0.00000000
## Hhold.fctrMKy:.clusterid.fctr2            0.00000000   0.00000000
## Hhold.fctrPKn:.clusterid.fctr2            0.00000000   0.00000000
## Hhold.fctrPKy:.clusterid.fctr2            0.00000000   0.00000000
## Hhold.fctrSKn:.clusterid.fctr2            0.00000000   0.00000000
## Hhold.fctrSKy:.clusterid.fctr2            0.00000000   0.00000000
## Hhold.fctrN:.clusterid.fctr3              0.00000000   0.00000000
## Hhold.fctrMKn:.clusterid.fctr3            0.00000000   0.00000000
## Hhold.fctrMKy:.clusterid.fctr3            0.00000000   0.00000000
## Hhold.fctrPKn:.clusterid.fctr3            0.00000000   0.00000000
## Hhold.fctrPKy:.clusterid.fctr3            0.00000000   0.00000000
## Hhold.fctrSKn:.clusterid.fctr3            0.00000000   0.00000000
## Hhold.fctrSKy:.clusterid.fctr3            0.00000000   0.00000000
## Hhold.fctrN:.clusterid.fctr4              0.00000000   0.00000000
## Hhold.fctrMKn:.clusterid.fctr4            0.00000000   0.00000000
## Hhold.fctrMKy:.clusterid.fctr4            0.00000000   0.00000000
## Hhold.fctrPKn:.clusterid.fctr4            0.00000000   0.00000000
## Hhold.fctrPKy:.clusterid.fctr4            0.00000000   0.00000000
## Hhold.fctrSKn:.clusterid.fctr4            0.00000000   0.00000000
## Hhold.fctrSKy:.clusterid.fctr4            0.00000000   0.00000000

#mdl_id <-"RFE.X.glmnet"; glb_featsimp_df <- myget_feats_importance(glb_models_lst[[mdl_id]], glb_featsimp_df); glb_featsimp_df[, paste0(mdl_id, ".imp")] <- glb_featsimp_df$imp; print(glb_featsimp_df)
#print(head(sbst_featsimp_df <- subset(glb_featsimp_df, is.na(RFE.X.glmnet.imp) | (abs(RFE.X.YeoJohnson.glmnet.imp - RFE.X.glmnet.imp) > 0.0001), select=-imp)))
#print(orderBy(~ -cor.y.abs, subset(glb_feats_df, id %in% c(row.names(sbst_featsimp_df), "startprice.dcm1.is9", "D.weight.post.stop.sum"))))

# Used again in fit.data.training & predict.data.new chunks
glb_analytics_diag_plots <- function(obs_df, mdl_id, prob_threshold=NULL) {
    if (!is.null(featsimp_df <- glb_featsimp_df)) {
        featsimp_df$feat <- gsub("`(.*?)`", "\\1", row.names(featsimp_df))    
        featsimp_df$feat.interact <- gsub("(.*?):(.*)", "\\2", featsimp_df$feat)
        featsimp_df$feat <- gsub("(.*?):(.*)", "\\1", featsimp_df$feat)    
        featsimp_df$feat.interact <- 
            ifelse(featsimp_df$feat.interact == featsimp_df$feat, 
                                            NA, featsimp_df$feat.interact)
        featsimp_df$feat <- 
            gsub("(.*?)\\.fctr(.*)", "\\1\\.fctr", featsimp_df$feat)
        featsimp_df$feat.interact <- 
            gsub("(.*?)\\.fctr(.*)", "\\1\\.fctr", featsimp_df$feat.interact) 
        featsimp_df <- orderBy(~ -imp.max, 
            summaryBy(imp ~ feat + feat.interact, data=featsimp_df,
                      FUN=max))    
        #rex_str=":(.*)"; txt_vctr=tail(featsimp_df$feat); ret_lst <- regexec(rex_str, txt_vctr); ret_lst <- regmatches(txt_vctr, ret_lst); ret_vctr <- sapply(1:length(ret_lst), function(pos_ix) ifelse(length(ret_lst[[pos_ix]]) > 0, ret_lst[[pos_ix]], "")); print(ret_vctr <- ret_vctr[ret_vctr != ""])    
        
        featsimp_df <- subset(featsimp_df, !is.na(imp.max))
        if (nrow(featsimp_df) > 5) {
            warning("Limiting important feature scatter plots to 5 out of ",
                    nrow(featsimp_df))
            featsimp_df <- head(featsimp_df, 5)
        }
        
    #     if (!all(is.na(featsimp_df$feat.interact)))
    #         stop("not implemented yet")
        rsp_var_out <- mygetPredictIds(glb_rsp_var, mdl_id)$value
        for (var in featsimp_df$feat) {
            plot_df <- melt(obs_df, id.vars = var, 
                            measure.vars = c(glb_rsp_var, rsp_var_out))
    
            print(myplot_scatter(plot_df, var, "value", colorcol_name = "variable",
                                facet_colcol_name = "variable", jitter = TRUE) + 
                          guides(color = FALSE))
        }
    }
    
    if (glb_is_regression) {
        if (is.null(featsimp_df) || (nrow(featsimp_df) == 0))
            warning("No important features in glb_fin_mdl") else
            print(myplot_prediction_regression(df=obs_df, 
                        feat_x=ifelse(nrow(featsimp_df) > 1, featsimp_df$feat[2],
                                      ".rownames"), 
                                               feat_y=featsimp_df$feat[1],
                        rsp_var=glb_rsp_var, rsp_var_out=rsp_var_out,
                        id_vars=glbFeatsId)
    #               + facet_wrap(reformulate(featsimp_df$feat[2])) # if [1 or 2] is a factor
    #               + geom_point(aes_string(color="<col_name>.fctr")) #  to color the plot
                  )
    }    
    
    if (glb_is_classification) {
        require(lazyeval)
        
        if (is.null(featsimp_df) || (nrow(featsimp_df) == 0))
            warning("No features in selected model are statistically important")
        else print(myplot_prediction_classification(df = obs_df, 
                                feat_x = ifelse(nrow(featsimp_df) > 1, 
                                                featsimp_df$feat[2], ".rownames"),
                                               feat_y = featsimp_df$feat[1],
                                                rsp_var = glb_rsp_var, 
                                                rsp_var_out = rsp_var_out, 
                                                id_vars = glbFeatsId,
                                                prob_threshold = prob_threshold))
    }    
}

if (glb_is_classification && glb_is_binomial)
    glb_analytics_diag_plots(obs_df = glbObsOOB, mdl_id = glbMdlSelId, 
            prob_threshold = glb_models_df[glb_models_df$id == glbMdlSelId, 
                                           "opt.prob.threshold.OOB"]) else
    glb_analytics_diag_plots(obs_df = glbObsOOB, mdl_id = glbMdlSelId)

## Warning in glb_analytics_diag_plots(obs_df = glbObsOOB, mdl_id =
## glbMdlSelId, : Limiting important feature scatter plots to 5 out of 108

## [1] "Min/Max Boundaries: "
## [1] "Inaccurate: "
##   USER_ID Party.fctr Party.fctr.All.X..rcv.glmnet.prob
## 1     943          D                         0.3112901
## 2    1393          D                         0.3191111
## 3    2798          D                         0.3202234
## 4    5028          D                         0.3268617
## 5    1843          D                         0.3293382
## 6    1045          D                         0.3293487
##   Party.fctr.All.X..rcv.glmnet Party.fctr.All.X..rcv.glmnet.err
## 1                            R                             TRUE
## 2                            R                             TRUE
## 3                            R                             TRUE
## 4                            R                             TRUE
## 5                            R                             TRUE
## 6                            R                             TRUE
##   Party.fctr.All.X..rcv.glmnet.err.abs Party.fctr.All.X..rcv.glmnet.is.acc
## 1                            0.6887099                               FALSE
## 2                            0.6808889                               FALSE
## 3                            0.6797766                               FALSE
## 4                            0.6731383                               FALSE
## 5                            0.6706618                               FALSE
## 6                            0.6706513                               FALSE
##   Party.fctr.All.X..rcv.glmnet.accurate Party.fctr.All.X..rcv.glmnet.error
## 1                                 FALSE                         -0.3887099
## 2                                 FALSE                         -0.3808889
## 3                                 FALSE                         -0.3797766
## 4                                 FALSE                         -0.3731383
## 5                                 FALSE                         -0.3706618
## 6                                 FALSE                         -0.3706513
##     USER_ID Party.fctr Party.fctr.All.X..rcv.glmnet.prob
## 12     3686          D                         0.3459625
## 106    4514          D                         0.4621860
## 206    6042          D                         0.5240140
## 250    1938          D                         0.5295261
## 307    6883          D                         0.5433422
## 449    1739          D                         0.6368048
##     Party.fctr.All.X..rcv.glmnet Party.fctr.All.X..rcv.glmnet.err
## 12                             R                             TRUE
## 106                            R                             TRUE
## 206                            R                             TRUE
## 250                            R                             TRUE
## 307                            R                             TRUE
## 449                            R                             TRUE
##     Party.fctr.All.X..rcv.glmnet.err.abs
## 12                             0.6540375
## 106                            0.5378140
## 206                            0.4759860
## 250                            0.4704739
## 307                            0.4566578
## 449                            0.3631952
##     Party.fctr.All.X..rcv.glmnet.is.acc
## 12                                FALSE
## 106                               FALSE
## 206                               FALSE
## 250                               FALSE
## 307                               FALSE
## 449                               FALSE
##     Party.fctr.All.X..rcv.glmnet.accurate
## 12                                  FALSE
## 106                                 FALSE
## 206                                 FALSE
## 250                                 FALSE
## 307                                 FALSE
## 449                                 FALSE
##     Party.fctr.All.X..rcv.glmnet.error
## 12                         -0.35403749
## 106                        -0.23781405
## 206                        -0.17598599
## 250                        -0.17047387
## 307                        -0.15665784
## 449                        -0.06319525
##     USER_ID Party.fctr Party.fctr.All.X..rcv.glmnet.prob
## 494    5466          R                         0.7868650
## 495     451          R                         0.7931354
## 496    1564          R                         0.7986553
## 497    3921          R                         0.7999019
## 498    4325          R                         0.8037402
## 499    1307          R                         0.8090310
##     Party.fctr.All.X..rcv.glmnet Party.fctr.All.X..rcv.glmnet.err
## 494                            D                             TRUE
## 495                            D                             TRUE
## 496                            D                             TRUE
## 497                            D                             TRUE
## 498                            D                             TRUE
## 499                            D                             TRUE
##     Party.fctr.All.X..rcv.glmnet.err.abs
## 494                            0.7868650
## 495                            0.7931354
## 496                            0.7986553
## 497                            0.7999019
## 498                            0.8037402
## 499                            0.8090310
##     Party.fctr.All.X..rcv.glmnet.is.acc
## 494                               FALSE
## 495                               FALSE
## 496                               FALSE
## 497                               FALSE
## 498                               FALSE
## 499                               FALSE
##     Party.fctr.All.X..rcv.glmnet.accurate
## 494                                 FALSE
## 495                                 FALSE
## 496                                 FALSE
## 497                                 FALSE
## 498                                 FALSE
## 499                                 FALSE
##     Party.fctr.All.X..rcv.glmnet.error
## 494                         0.08686496
## 495                         0.09313539
## 496                         0.09865532
## 497                         0.09990189
## 498                         0.10374022
## 499                         0.10903104

if (!is.null(glbFeatsCategory)) {
    glbLvlCategory <- merge(glbLvlCategory, 
            myget_category_stats(obs_df = glbObsFit, mdl_id = glbMdlSelId, 
                                 label = "fit"), 
                            by = glbFeatsCategory, all = TRUE)
    row.names(glbLvlCategory) <- glbLvlCategory[, glbFeatsCategory]
    glbLvlCategory <- merge(glbLvlCategory, 
            myget_category_stats(obs_df = glbObsOOB, mdl_id = glbMdlSelId,
                                 label="OOB"),
                          #by=glbFeatsCategory, all=TRUE) glb_ctgry-df already contains .n.OOB ?
                          all = TRUE)
    row.names(glbLvlCategory) <- glbLvlCategory[, glbFeatsCategory]
    if (any(grepl("OOB", glbMdlMetricsEval)))
        print(orderBy(~-err.abs.OOB.mean, glbLvlCategory)) else
            print(orderBy(~-err.abs.fit.mean, glbLvlCategory))
    print(colSums(glbLvlCategory[, -grep(glbFeatsCategory, names(glbLvlCategory))]))
}

##     Hhold.fctr .n.OOB .n.Fit .n.Tst .freqRatio.Fit .freqRatio.OOB
## PKy        PKy      9     52     10     0.01169065    0.008035714
## PKn        PKn     30    150     37     0.03372302    0.026785714
## N            N     83    367    102     0.08250899    0.074107143
## SKn        SKn    511   1920    638     0.43165468    0.456250000
## SKy        SKy     53    147     65     0.03304856    0.047321429
## MKn        MKn    136    516    169     0.11600719    0.121428571
## MKy        MKy    298   1296    371     0.29136691    0.266071429
##     .freqRatio.Tst err.abs.fit.sum err.abs.fit.mean .n.fit err.abs.OOB.sum
## PKy    0.007183908        24.71367        0.4752630     52         4.28726
## PKn    0.026580460        59.07825        0.3938550    150        14.10653
## N      0.073275862       173.80353        0.4735791    367        38.82857
## SKn    0.458333333       884.82425        0.4608460   1920       238.72729
## SKy    0.046695402        65.43207        0.4451161    147        24.55390
## MKn    0.121408046       235.21548        0.4558439    516        62.21564
## MKy    0.266522989       595.03188        0.4591295   1296       134.54661
##     err.abs.OOB.mean
## PKy        0.4763622
## PKn        0.4702178
## N          0.4678140
## SKn        0.4671767
## SKy        0.4632812
## MKn        0.4574680
## MKy        0.4514987
##           .n.OOB           .n.Fit           .n.Tst   .freqRatio.Fit 
##      1120.000000      4448.000000      1392.000000         1.000000 
##   .freqRatio.OOB   .freqRatio.Tst  err.abs.fit.sum err.abs.fit.mean 
##         1.000000         1.000000      2038.099138         3.163633 
##           .n.fit  err.abs.OOB.sum err.abs.OOB.mean 
##      4448.000000       517.265807         3.253819

write.csv(glbObsOOB[, c(glbFeatsId, 
                grep(glb_rsp_var, names(glbObsOOB), fixed=TRUE, value=TRUE))], 
    paste0(gsub(".", "_", paste0(glbOut$pfx, glbMdlSelId), fixed=TRUE), 
           "_OOBobs.csv"), row.names=FALSE)

fit.models_2_chunk_df <- 
    myadd_chunk(NULL, "fit.models_2_bgn", label.minor = "teardown")

##              label step_major step_minor label_minor     bgn end elapsed
## 1 fit.models_2_bgn          1          0    teardown 429.036  NA      NA

glb_chunks_df <- myadd_chunk(glb_chunks_df, "fit.models", major.inc=FALSE)

##        label step_major step_minor label_minor     bgn     end elapsed
## 6 fit.models          4          2           2 415.955 429.046  13.092
## 7 fit.models          4          3           3 429.047      NA      NA

# if (sum(is.na(glbObsAll$D.P.http)) > 0)
#         stop("fit.models_3: Why is this happening ?")

#stop(here"); glb2Sav()
sync_glb_obs_df <- function() {
    # Merge or cbind ?
    for (col in setdiff(names(glbObsFit), names(glbObsTrn)))
        glbObsTrn[glbObsTrn$.lcn == "Fit", col] <<- glbObsFit[, col]
    for (col in setdiff(names(glbObsFit), names(glbObsAll)))
        glbObsAll[glbObsAll$.lcn == "Fit", col] <<- glbObsFit[, col]
    if (all(is.na(glbObsNew[, glb_rsp_var])))
        for (col in setdiff(names(glbObsOOB), names(glbObsTrn)))
            glbObsTrn[glbObsTrn$.lcn == "OOB", col] <<- glbObsOOB[, col]
    for (col in setdiff(names(glbObsOOB), names(glbObsAll)))
        glbObsAll[glbObsAll$.lcn == "OOB", col] <<- glbObsOOB[, col]
}
sync_glb_obs_df()
    
print(setdiff(names(glbObsNew), names(glbObsAll)))

## character(0)

replay.petrisim(pn = glb_analytics_pn, 
    replay.trans = (glb_analytics_avl_objs <- c(glb_analytics_avl_objs, 
        "model.selected")), flip_coord = TRUE)

## time trans    "bgn " "fit.data.training.all " "predict.data.new " "end " 
## 0.0000   multiple enabled transitions:  data.training.all data.new model.selected    firing:  model.selected 
## 1.0000    3   2 1 0 0

glb_chunks_df <- myadd_chunk(glb_chunks_df, "fit.data.training", major.inc = TRUE)

##               label step_major step_minor label_minor     bgn    end
## 7        fit.models          4          3           3 429.047 433.74
## 8 fit.data.training          5          0           0 433.741     NA
##   elapsed
## 7   4.693
## 8      NA

Step `5.0: fit data training`

#load(paste0(glb_inp_pfx, "dsk.RData"))

if (!is.null(glbMdlFinId) && (glbMdlFinId %in% names(glb_models_lst))) {
    warning("Final model same as user selected model")
    glb_fin_mdl <- glb_models_lst[[glbMdlFinId]]
} else 
# if (nrow(glbObsFit) + length(glbObsFitOutliers) == nrow(glbObsTrn))
if (!all(is.na(glbObsNew[, glb_rsp_var])))
{    
    warning("Final model same as glbMdlSelId")
    glbMdlFinId <- paste0("Final.", glbMdlSelId)
    glb_fin_mdl <- glb_sel_mdl
    glb_models_lst[[glbMdlFinId]] <- glb_fin_mdl
    mdlDf <- glb_models_df[glb_models_df$id == glbMdlSelId, ]
    mdlDf$id <- glbMdlFinId
    glb_models_df <- rbind(glb_models_df, mdlDf)
} else {    
            if (grepl("RFE\\.X", names(glbMdlFamilies))) {
                indepVar <- mygetIndepVar(glb_feats_df)
                rfe_trn_results <- 
                    myrun_rfe(glbObsTrn, indepVar, glbRFESizes[["Final"]])
                if (!isTRUE(all.equal(sort(predictors(rfe_trn_results)),
                                      sort(predictors(rfe_fit_results))))) {
                    print("Diffs predictors(rfe_trn_results) vs. predictors(rfe_fit_results):")
                    print(setdiff(predictors(rfe_trn_results), predictors(rfe_fit_results)))
                    print("Diffs predictors(rfe_fit_results) vs. predictors(rfe_trn_results):")
                    print(setdiff(predictors(rfe_fit_results), predictors(rfe_trn_results)))
            }
        }
    # }    

    if (grepl("Ensemble", glbMdlSelId)) {
        # Find which models are relevant
        mdlimp_df <- subset(myget_feats_importance(glb_sel_mdl), imp > 5)
        # Fit selected models on glbObsTrn
        for (mdl_id in gsub(".prob", "", 
gsub(mygetPredictIds(glb_rsp_var)$value, "", row.names(mdlimp_df), fixed = TRUE),
                            fixed = TRUE)) {
            mdl_id_components <- unlist(strsplit(mdl_id, "[.]"))
            mdlIdPfx <- paste0(c(head(mdl_id_components, -1), "Train"), 
                               collapse = ".")
            if (grepl("RFE\\.X\\.", mdlIdPfx)) 
                mdlIndepVars <- myadjustInteractionFeats(glb_feats_df, myextract_actual_feats(
                    predictors(rfe_trn_results))) else
                mdlIndepVars <- trim(unlist(
            strsplit(glb_models_df[glb_models_df$id == mdl_id, "feats"], "[,]")))
            ret_lst <- 
                myfit_mdl(mdl_specs_lst = myinit_mdl_specs_lst(mdl_specs_lst = list(
                        id.prefix = mdlIdPfx, 
                        type = glb_model_type, tune.df = glbMdlTuneParams,
                        trainControl.method = "repeatedcv",
                        trainControl.number = glb_rcv_n_folds,
                        trainControl.repeats = glb_rcv_n_repeats,
                        trainControl.classProbs = glb_is_classification,
                        trainControl.summaryFunction = glbMdlMetricSummaryFn,
                        train.metric = glbMdlMetricSummary, 
                        train.maximize = glbMdlMetricMaximize,    
                        train.method = tail(mdl_id_components, 1))),
                    indepVar = mdlIndepVars,
                    rsp_var = glb_rsp_var, 
                    fit_df = glbObsTrn, OOB_df = NULL)
            
            glbObsTrn <- glb_get_predictions(df = glbObsTrn,
                                                mdl_id = tail(glb_models_df$id, 1), 
                                                rsp_var = glb_rsp_var,
                                                prob_threshold_def = 
                    subset(glb_models_df, id == mdl_id)$opt.prob.threshold.OOB)
            glbObsNew <- glb_get_predictions(df = glbObsNew,
                                                mdl_id = tail(glb_models_df$id, 1), 
                                                rsp_var = glb_rsp_var,
                                                prob_threshold_def = 
                    subset(glb_models_df, id == mdl_id)$opt.prob.threshold.OOB)
        }    
    }
    
    # "Final" model
    if ((model_method <- glb_sel_mdl$method) == "custom")
        # get actual method from the mdl_id
        model_method <- tail(unlist(strsplit(glbMdlSelId, "[.]")), 1)
        
    if (grepl("Ensemble", glbMdlSelId)) {
        # Find which models are relevant
        mdlimp_df <- subset(myget_feats_importance(glb_sel_mdl), imp > 5)
        if (glb_is_classification && glb_is_binomial)
            indepVar <- gsub("(.*)\\.(.*)\\.prob", "\\1\\.Train\\.\\2\\.prob",
                                    row.names(mdlimp_df)) else
            indepVar <- gsub("(.*)\\.(.*)", "\\1\\.Train\\.\\2",
                                    row.names(mdlimp_df))
    } else 
    if (grepl("RFE.X", glbMdlSelId, fixed = TRUE)) {
        indepVar <- myextract_actual_feats(predictors(rfe_trn_results))
    } else indepVar <- 
                trim(unlist(strsplit(glb_models_df[glb_models_df$id ==
                                                   glbMdlSelId
                                                   , "feats"], "[,]")))
        
    if (!is.null(glb_preproc_methods) &&
        ((match_pos <- regexpr(gsub(".", "\\.", 
                                    paste(glb_preproc_methods, collapse = "|"),
                                   fixed = TRUE), glbMdlSelId)) != -1))
        ths_preProcess <- str_sub(glbMdlSelId, match_pos, 
                                match_pos + attr(match_pos, "match.length") - 1) else
        ths_preProcess <- NULL                                      

    mdl_id_pfx <- ifelse(grepl("Ensemble", glbMdlSelId),
                                   "Final.Ensemble", "Final")
    
    trnobs_df <- glbObsTrn 
    if (!is.null(glbObsTrnOutliers[[mdl_id_pfx]])) {
        trnobs_df <- glbObsTrn[!(glbObsTrn[, glbFeatsId] %in% glbObsTrnOutliers[[mdl_id_pfx]]), ]
        print(sprintf("Outliers removed: %d", nrow(glbObsTrn) - nrow(trnobs_df)))
        print(setdiff(glbObsTrn[, glbFeatsId], trnobs_df[, glbFeatsId]))
    }    
        
    # Force fitting of Final.glm to identify outliers
    method_vctr <- unique(c(myparseMdlId(glbMdlSelId)$alg, glbMdlFamilies[["Final"]]))
    for (method in method_vctr) {
        #source("caret_nominalTrainWorkflow.R")
        
        # glmnet requires at least 2 indep vars
        if ((length(indepVar) == 1) && (method %in% "glmnet"))
            next
        
        ret_lst <- 
            myfit_mdl(mdl_specs_lst = myinit_mdl_specs_lst(mdl_specs_lst = list(
                    id.prefix = mdl_id_pfx, 
                    type = glb_model_type, trainControl.method = "repeatedcv",
                    trainControl.number = glb_rcv_n_folds, 
                    trainControl.repeats = glb_rcv_n_repeats,
                    trainControl.classProbs = glb_is_classification,
                    trainControl.summaryFunction = glbMdlMetricSummaryFn,
                    trainControl.allowParallel = glbMdlAllowParallel,
                    train.metric = glbMdlMetricSummary, 
                    train.maximize = glbMdlMetricMaximize,    
                    train.method = method,
                    train.preProcess = ths_preProcess)),
                indepVar = indepVar, rsp_var = glb_rsp_var, 
                fit_df = trnobs_df, OOB_df = NULL)
        
        if ((length(method_vctr) == 1) || (method != "glm")) {
            glb_fin_mdl <- glb_models_lst[[length(glb_models_lst)]] 
            glbMdlFinId <- glb_models_df[length(glb_models_lst), "id"]
        }
    }
        
}

## [1] "myfit_mdl: enter: 0.000000 secs"
## [1] "fitting model: Final##rcv#glmnet"
## [1] "    indepVar: Q109244.fctr,Hhold.fctr,Edn.fctr,Q101163.fctr,Q100689.fctr,Q98078.fctr,Q99716.fctr,Q120379.fctr,Q121699.fctr,Q105840.fctr,Q113583.fctr,Q115195.fctr,Q102089.fctr,Q98059.fctr,Q114386.fctr,Q100680.fctr,Q108342.fctr,Q111848.fctr,YOB.Age.fctr,Q118892.fctr,Q102687.fctr,Q115390.fctr,Q119851.fctr,Q114517.fctr,Q120012.fctr,Q109367.fctr,Q114961.fctr,Q121700.fctr,Q124122.fctr,Q111220.fctr,Q113992.fctr,Q121011.fctr,Q106042.fctr,Q116448.fctr,Q116601.fctr,Q104996.fctr,Q102906.fctr,Q113584.fctr,Q108950.fctr,Q102674.fctr,Q103293.fctr,Q112478.fctr,Q114748.fctr,Q107491.fctr,Q100562.fctr,Q108617.fctr,Q100010.fctr,Q115602.fctr,Q116953.fctr,Q115610.fctr,Q106997.fctr,Q120978.fctr,Q112512.fctr,Q108343.fctr,Q96024.fctr,Q106389.fctr,.rnorm,Q108754.fctr,Q98578.fctr,Q101162.fctr,Q115777.fctr,Q99581.fctr,Q124742.fctr,Q116797.fctr,Q112270.fctr,Q118237.fctr,Q119650.fctr,Q111580.fctr,Q123464.fctr,Q117193.fctr,Q99982.fctr,Q108856.fctr,Q118233.fctr,Q102289.fctr,Q116197.fctr,Income.fctr,Q118232.fctr,Q120194.fctr,Q114152.fctr,Q122770.fctr,Q117186.fctr,Q105655.fctr,Q106993.fctr,Q119334.fctr,Q122120.fctr,Q116441.fctr,Q118117.fctr,Q123621.fctr,Q122769.fctr,Q120650.fctr,Q98869.fctr,Q107869.fctr,Q120014.fctr,Q115899.fctr,Q106388.fctr,Q99480.fctr,Q122771.fctr,Q108855.fctr,Q110740.fctr,Q106272.fctr,Q101596.fctr,Q116881.fctr,Q120472.fctr,Q98197.fctr,Q113181.fctr,Q115611.fctr,Gender.fctr,Hhold.fctr:.clusterid.fctr"
## [1] "myfit_mdl: setup complete: 0.730000 secs"
## Aggregating results
## Selecting tuning parameters
## Fitting alpha = 1, lambda = 0.0113 on full training set
## [1] "myfit_mdl: train complete: 28.273000 secs"

## Warning in myfit_mdl(mdl_specs_lst = myinit_mdl_specs_lst(mdl_specs_lst
## = list(id.prefix = mdl_id_pfx, : model's bestTune found at an extreme of
## tuneGrid for parameter: alpha

##             Length Class      Mode     
## a0             76  -none-     numeric  
## beta        19228  dgCMatrix  S4       
## df             76  -none-     numeric  
## dim             2  -none-     numeric  
## lambda         76  -none-     numeric  
## dev.ratio      76  -none-     numeric  
## nulldev         1  -none-     numeric  
## npasses         1  -none-     numeric  
## jerr            1  -none-     numeric  
## offset          1  -none-     logical  
## classnames      2  -none-     character
## call            5  -none-     call     
## nobs            1  -none-     numeric  
## lambdaOpt       1  -none-     numeric  
## xNames        253  -none-     character
## problemType     1  -none-     character
## tuneValue       2  data.frame list     
## obsLevels       2  -none-     character
## [1] "min lambda > lambdaOpt:"
##             (Intercept)              Edn.fctr.L            Gender.fctrM 
##             0.194438500             0.005324127            -0.103324489 
##           Hhold.fctrMKy           Hhold.fctrPKn           Income.fctr.Q 
##            -0.041820884             0.337598807            -0.031826828 
##           Income.fctr.C         Q100689.fctrYes         Q101163.fctrDad 
##            -0.024094689             0.046302965            -0.085936176 
##         Q101163.fctrMom          Q106042.fctrNo          Q106997.fctrGr 
##             0.033483931            -0.016322442            -0.039899178 
##        Q108855.fctrYes!          Q109244.fctrNo         Q109244.fctrYes 
##            -0.001342055            -0.351878919             0.927553120 
##          Q110740.fctrPC          Q112478.fctrNo          Q113181.fctrNo 
##            -0.063633328            -0.004486470             0.084350998 
##         Q113181.fctrYes         Q115390.fctrYes          Q115611.fctrNo 
##            -0.150310848             0.028773879             0.135957434 
##         Q115611.fctrYes          Q115899.fctrCs       Q116881.fctrRight 
##            -0.314087372             0.020277788            -0.152866221 
##          Q118232.fctrId          Q119851.fctrNo Q120194.fctrStudy first 
##             0.065373988            -0.073935701             0.006842972 
##         Q120379.fctrYes     Q120472.fctrScience         Q121699.fctrYes 
##             0.062412204            -0.051347228             0.004660527 
##          Q122771.fctrPt           Q98197.fctrNo          Q98197.fctrYes 
##            -0.034676799             0.267730304            -0.004059434 
##           Q98869.fctrNo           Q99480.fctrNo          Q99480.fctrYes 
##             0.152390779             0.027812738            -0.016334564 
##          YOB.Age.fctr.L 
##             0.007621652 
## [1] "max lambda < lambdaOpt:"
##             (Intercept)              Edn.fctr.L            Gender.fctrM 
##            0.2005481210            0.0121867712           -0.1045038082 
##           Hhold.fctrMKy           Hhold.fctrPKn           Income.fctr.Q 
##           -0.0567296046            0.3536530688           -0.0417564097 
##           Income.fctr.C         Q100689.fctrYes         Q101163.fctrDad 
##           -0.0405868380            0.0572628028           -0.0898329179 
##         Q101163.fctrMom          Q104996.fctrNo          Q106042.fctrNo 
##            0.0404889559           -0.0040458520           -0.0204082825 
##          Q106389.fctrNo          Q106997.fctrGr        Q108855.fctrYes! 
##           -0.0019669304           -0.0526759378           -0.0105337529 
##          Q109244.fctrNo         Q109244.fctrYes         Q110740.fctrMac 
##           -0.3524375639            0.9325346035            0.0004260959 
##          Q110740.fctrPC          Q112478.fctrNo          Q113181.fctrNo 
##           -0.0714307603           -0.0136559492            0.0925851389 
##         Q113181.fctrYes         Q115390.fctrYes          Q115611.fctrNo 
##           -0.1472971203            0.0411038655            0.1363360518 
##         Q115611.fctrYes          Q115899.fctrCs       Q116881.fctrRight 
##           -0.3214640701            0.0308046421           -0.1629024808 
##          Q116953.fctrNo          Q118232.fctrId          Q119851.fctrNo 
##           -0.0067662252            0.0776017956           -0.0850316406 
## Q120194.fctrStudy first          Q120379.fctrNo         Q120379.fctrYes 
##            0.0189244856           -0.0009969328            0.0733627149 
##     Q120472.fctrScience         Q121699.fctrYes          Q122771.fctrPt 
##           -0.0630476692            0.0140702837           -0.0494428363 
##           Q98197.fctrNo          Q98197.fctrYes           Q98869.fctrNo 
##            0.2766127756           -0.0015949896            0.1651846600 
##           Q99480.fctrNo          Q99480.fctrYes          YOB.Age.fctr.L 
##            0.0301079760           -0.0287615105            0.0265444403 
##          YOB.Age.fctr^8 
##           -0.0060255057 
## [1] "myfit_mdl: train diagnostics complete: 28.948000 secs"

##          Prediction
## Reference    R    D
##         R 2342  275
##         D 1965  986
##       Accuracy          Kappa  AccuracyLower  AccuracyUpper   AccuracyNull 
##   5.977011e-01   2.209666e-01   5.846829e-01   6.106169e-01   5.299928e-01 
## AccuracyPValue  McnemarPValue 
##   1.614933e-24  6.371336e-279 
## [1] "myfit_mdl: predict complete: 36.618000 secs"
##                  id
## 1 Final##rcv#glmnet
##                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               feats
## 1 Q109244.fctr,Hhold.fctr,Edn.fctr,Q101163.fctr,Q100689.fctr,Q98078.fctr,Q99716.fctr,Q120379.fctr,Q121699.fctr,Q105840.fctr,Q113583.fctr,Q115195.fctr,Q102089.fctr,Q98059.fctr,Q114386.fctr,Q100680.fctr,Q108342.fctr,Q111848.fctr,YOB.Age.fctr,Q118892.fctr,Q102687.fctr,Q115390.fctr,Q119851.fctr,Q114517.fctr,Q120012.fctr,Q109367.fctr,Q114961.fctr,Q121700.fctr,Q124122.fctr,Q111220.fctr,Q113992.fctr,Q121011.fctr,Q106042.fctr,Q116448.fctr,Q116601.fctr,Q104996.fctr,Q102906.fctr,Q113584.fctr,Q108950.fctr,Q102674.fctr,Q103293.fctr,Q112478.fctr,Q114748.fctr,Q107491.fctr,Q100562.fctr,Q108617.fctr,Q100010.fctr,Q115602.fctr,Q116953.fctr,Q115610.fctr,Q106997.fctr,Q120978.fctr,Q112512.fctr,Q108343.fctr,Q96024.fctr,Q106389.fctr,.rnorm,Q108754.fctr,Q98578.fctr,Q101162.fctr,Q115777.fctr,Q99581.fctr,Q124742.fctr,Q116797.fctr,Q112270.fctr,Q118237.fctr,Q119650.fctr,Q111580.fctr,Q123464.fctr,Q117193.fctr,Q99982.fctr,Q108856.fctr,Q118233.fctr,Q102289.fctr,Q116197.fctr,Income.fctr,Q118232.fctr,Q120194.fctr,Q114152.fctr,Q122770.fctr,Q117186.fctr,Q105655.fctr,Q106993.fctr,Q119334.fctr,Q122120.fctr,Q116441.fctr,Q118117.fctr,Q123621.fctr,Q122769.fctr,Q120650.fctr,Q98869.fctr,Q107869.fctr,Q120014.fctr,Q115899.fctr,Q106388.fctr,Q99480.fctr,Q122771.fctr,Q108855.fctr,Q110740.fctr,Q106272.fctr,Q101596.fctr,Q116881.fctr,Q120472.fctr,Q98197.fctr,Q113181.fctr,Q115611.fctr,Gender.fctr,Hhold.fctr:.clusterid.fctr
##   max.nTuningRuns min.elapsedtime.everything min.elapsedtime.final
## 1              25                     27.408                 2.424
##   max.AUCpROC.fit max.Sens.fit max.Spec.fit max.AUCROCR.fit
## 1       0.6362495    0.5483378    0.7241613       0.2991549
##   opt.prob.threshold.fit max.f.score.fit max.Accuracy.fit
## 1                    0.6       0.6764876        0.6329006
##   max.AccuracyLower.fit max.AccuracyUpper.fit max.Kappa.fit
## 1             0.5846829             0.6106169     0.2589576
##   max.AccuracySD.fit max.KappaSD.fit
## 1         0.01176231      0.02395599
## [1] "myfit_mdl: exit: 36.635000 secs"

rm(ret_lst)
glb_chunks_df <- myadd_chunk(glb_chunks_df, "fit.data.training", major.inc=FALSE)

##               label step_major step_minor label_minor     bgn     end
## 8 fit.data.training          5          0           0 433.741 470.985
## 9 fit.data.training          5          1           1 470.986      NA
##   elapsed
## 8  37.244
## 9      NA

#stop(here"); glb2Sav()
if (glb_is_classification && glb_is_binomial) 
    prob_threshold <- glb_models_df[glb_models_df$id == glbMdlSelId,
                                        "opt.prob.threshold.OOB"] else 
    prob_threshold <- NULL

if (grepl("Ensemble", glbMdlFinId)) {
    # Get predictions for each model in ensemble; Outliers that have been moved to OOB might not have been predicted yet
    mdlEnsembleComps <- unlist(str_split(subset(glb_models_df, 
                                                id == glbMdlFinId)$feats, ","))
    if (glb_is_classification && glb_is_binomial)
        mdlEnsembleComps <- gsub("\\.prob$", "", mdlEnsembleComps)
    mdlEnsembleComps <- gsub(paste0("^", 
                        gsub(".", "\\.", mygetPredictIds(glb_rsp_var)$value, fixed = TRUE)),
                             "", mdlEnsembleComps)
    for (mdl_id in mdlEnsembleComps) {
        glbObsTrn <- glb_get_predictions(df = glbObsTrn, mdl_id = mdl_id, 
                                            rsp_var = glb_rsp_var,
                                            prob_threshold_def = prob_threshold)
        glbObsNew <- glb_get_predictions(df = glbObsNew, mdl_id = mdl_id, 
                                            rsp_var = glb_rsp_var,
                                            prob_threshold_def = prob_threshold)
    }    
}
glbObsTrn <- glb_get_predictions(df = glbObsTrn, mdl_id = glbMdlFinId, 
                                     rsp_var = glb_rsp_var,
                                    prob_threshold_def = prob_threshold)

## Warning in glb_get_predictions(df = glbObsTrn, mdl_id = glbMdlFinId,
## rsp_var = glb_rsp_var, : Using default probability threshold: 0.7

glb_featsimp_df <- myget_feats_importance(mdl=glb_fin_mdl,
                                          featsimp_df=glb_featsimp_df)
#glb_featsimp_df[, paste0(glbMdlFinId, ".imp")] <- glb_featsimp_df$imp
print(glb_featsimp_df)

##                                All.X..rcv.glmnet.imp Final..rcv.glmnet.imp
## Q109244.fctrYes                         100.00000000          100.00000000
## Q109244.fctrNo                           39.80933610           37.85598680
## Hhold.fctrPKn                            41.56642724           37.25545617
## Q115611.fctrYes                          38.97373582           34.20502961
## Q98197.fctrNo                            20.63341630           29.31306049
## Q98869.fctrNo                            20.36490400           17.15145611
## Q116881.fctrRight                        10.34339619           17.03627425
## Q113181.fctrYes                          10.03215580           15.97468665
## Q115611.fctrNo                            9.69659664           14.63644538
## Gender.fctrM                              7.31671883           11.17712069
## Q113181.fctrNo                            4.47059621            9.56313129
## Q101163.fctrDad                           5.93777650            9.47197006
## Q119851.fctrNo                            3.95883221            8.61619297
## Q118232.fctrId                            2.50949804            7.76417484
## Q120379.fctrYes                           3.59678995            7.36880104
## Q110740.fctrPC                            1.70814173            7.30992383
## Q120472.fctrScience                       0.00000000            6.22468445
## Q100689.fctrYes                           0.00000000            5.63783468
## Hhold.fctrMKy                             8.30653632            5.39418970
## Q106997.fctrGr                            0.00000000            5.05907385
## Q122771.fctrPt                            0.01539812            4.61769175
## Q101163.fctrMom                           4.60717672            4.02148246
## Income.fctr.Q                             0.05099013            4.01971777
## Q115390.fctrYes                           0.00000000            3.83631167
## Income.fctr.C                             0.00000000            3.58434097
## Q99480.fctrNo                             7.40277198            3.12790282
## Q115899.fctrCs                            0.01891999            2.81436544
## Q99480.fctrYes                            0.00000000            2.50509763
## Q106042.fctrNo                            0.00000000            2.00082292
## YOB.Age.fctr.L                            0.00000000            1.96027716
## Q120194.fctrStudy first                   0.00000000            1.46404807
## Q121699.fctrYes                           0.00000000            1.06835640
## Q112478.fctrNo                            0.00000000            1.03515874
## Edn.fctr.L                                0.00000000            0.98609275
## Q108855.fctrYes!                          0.00000000            0.69851577
## Q116953.fctrNo                            0.00000000            0.40800584
## YOB.Age.fctr^8                            0.00000000            0.36334019
## Q98197.fctrYes                            4.27686282            0.28772824
## Q104996.fctrNo                            0.00000000            0.24396635
## Q106389.fctrNo                            0.00000000            0.11860662
## Q120379.fctrNo                            0.14882265            0.06011541
## Q110740.fctrMac                           0.00000000            0.02569374
## .rnorm                                    0.00000000            0.00000000
## Edn.fctr.C                                0.00000000            0.00000000
## Edn.fctr.Q                                0.00000000            0.00000000
## Edn.fctr^4                                0.00000000            0.00000000
## Edn.fctr^5                                0.00000000            0.00000000
## Edn.fctr^6                                0.00000000            0.00000000
## Edn.fctr^7                                0.00000000            0.00000000
## Gender.fctrF                              0.00000000            0.00000000
## Hhold.fctrMKn                             0.00000000            0.00000000
## Hhold.fctrMKn:.clusterid.fctr2            0.00000000            0.00000000
## Hhold.fctrMKn:.clusterid.fctr3            0.00000000            0.00000000
## Hhold.fctrMKn:.clusterid.fctr4            0.00000000            0.00000000
## Hhold.fctrMKy:.clusterid.fctr2            0.00000000            0.00000000
## Hhold.fctrMKy:.clusterid.fctr3            0.00000000            0.00000000
## Hhold.fctrMKy:.clusterid.fctr4            0.00000000            0.00000000
## Hhold.fctrN:.clusterid.fctr2              0.00000000            0.00000000
## Hhold.fctrN:.clusterid.fctr3              0.00000000            0.00000000
## Hhold.fctrN:.clusterid.fctr4              0.00000000            0.00000000
## Hhold.fctrPKn:.clusterid.fctr2            0.00000000            0.00000000
## Hhold.fctrPKn:.clusterid.fctr3            0.00000000            0.00000000
## Hhold.fctrPKn:.clusterid.fctr4            0.00000000            0.00000000
## Hhold.fctrPKy                             0.00000000            0.00000000
## Hhold.fctrPKy:.clusterid.fctr2            0.00000000            0.00000000
## Hhold.fctrPKy:.clusterid.fctr3            0.00000000            0.00000000
## Hhold.fctrPKy:.clusterid.fctr4            0.00000000            0.00000000
## Hhold.fctrSKn                             0.00000000            0.00000000
## Hhold.fctrSKn:.clusterid.fctr2            0.00000000            0.00000000
## Hhold.fctrSKn:.clusterid.fctr3            0.00000000            0.00000000
## Hhold.fctrSKn:.clusterid.fctr4            0.00000000            0.00000000
## Hhold.fctrSKy                             0.00000000            0.00000000
## Hhold.fctrSKy:.clusterid.fctr2            0.00000000            0.00000000
## Hhold.fctrSKy:.clusterid.fctr3            0.00000000            0.00000000
## Hhold.fctrSKy:.clusterid.fctr4            0.00000000            0.00000000
## Income.fctr.L                             0.00000000            0.00000000
## Income.fctr^4                             0.00000000            0.00000000
## Income.fctr^5                             0.00000000            0.00000000
## Income.fctr^6                             0.00000000            0.00000000
## Q100010.fctrNo                            0.00000000            0.00000000
## Q100010.fctrYes                           0.00000000            0.00000000
## Q100562.fctrNo                            0.00000000            0.00000000
## Q100562.fctrYes                           0.00000000            0.00000000
## Q100680.fctrNo                            0.00000000            0.00000000
## Q100680.fctrYes                           0.00000000            0.00000000
## Q100689.fctrNo                            0.00000000            0.00000000
## Q101162.fctrOptimist                      0.00000000            0.00000000
## Q101162.fctrPessimist                     0.00000000            0.00000000
## Q101596.fctrNo                            0.00000000            0.00000000
## Q101596.fctrYes                           0.00000000            0.00000000
## Q102089.fctrOwn                           0.00000000            0.00000000
## Q102089.fctrRent                          0.00000000            0.00000000
## Q102289.fctrNo                            0.00000000            0.00000000
## Q102289.fctrYes                           0.00000000            0.00000000
## Q102674.fctrNo                            0.00000000            0.00000000
## Q102674.fctrYes                           0.00000000            0.00000000
## Q102687.fctrNo                            0.00000000            0.00000000
## Q102687.fctrYes                           0.00000000            0.00000000
## Q102906.fctrNo                            0.00000000            0.00000000
## Q102906.fctrYes                           0.00000000            0.00000000
## Q103293.fctrNo                            0.00000000            0.00000000
## Q103293.fctrYes                           0.00000000            0.00000000
## Q104996.fctrYes                           0.00000000            0.00000000
## Q105655.fctrNo                            0.00000000            0.00000000
## Q105655.fctrYes                           0.00000000            0.00000000
## Q105840.fctrNo                            0.00000000            0.00000000
## Q105840.fctrYes                           0.00000000            0.00000000
## Q106042.fctrYes                           0.00000000            0.00000000
## Q106272.fctrNo                            0.00000000            0.00000000
## Q106272.fctrYes                           0.00000000            0.00000000
## Q106388.fctrNo                            0.00000000            0.00000000
## Q106388.fctrYes                           0.00000000            0.00000000
## Q106389.fctrYes                           0.00000000            0.00000000
## Q106993.fctrNo                            0.00000000            0.00000000
## Q106993.fctrYes                           0.00000000            0.00000000
## Q106997.fctrYy                            0.00000000            0.00000000
## Q107491.fctrNo                            0.00000000            0.00000000
## Q107491.fctrYes                           0.00000000            0.00000000
## Q107869.fctrNo                            0.00000000            0.00000000
## Q107869.fctrYes                           0.00000000            0.00000000
## Q108342.fctrIn-person                     0.00000000            0.00000000
## Q108342.fctrOnline                        0.00000000            0.00000000
## Q108343.fctrNo                            0.00000000            0.00000000
## Q108343.fctrYes                           0.00000000            0.00000000
## Q108617.fctrNo                            0.00000000            0.00000000
## Q108617.fctrYes                           0.00000000            0.00000000
## Q108754.fctrNo                            0.00000000            0.00000000
## Q108754.fctrYes                           0.00000000            0.00000000
## Q108855.fctrUmm...                        0.00000000            0.00000000
## Q108856.fctrSocialize                     0.00000000            0.00000000
## Q108856.fctrSpace                         0.00000000            0.00000000
## Q108950.fctrCautious                      0.00000000            0.00000000
## Q108950.fctrRisk-friendly                 0.00000000            0.00000000
## Q109367.fctrNo                            0.00000000            0.00000000
## Q109367.fctrYes                           0.00000000            0.00000000
## Q111220.fctrNo                            0.00000000            0.00000000
## Q111220.fctrYes                           0.00000000            0.00000000
## Q111580.fctrDemanding                     0.00000000            0.00000000
## Q111580.fctrSupportive                    0.00000000            0.00000000
## Q111848.fctrNo                            0.00000000            0.00000000
## Q111848.fctrYes                           0.00000000            0.00000000
## Q112270.fctrNo                            0.00000000            0.00000000
## Q112270.fctrYes                           0.00000000            0.00000000
## Q112478.fctrYes                           0.00000000            0.00000000
## Q112512.fctrNo                            0.00000000            0.00000000
## Q112512.fctrYes                           0.00000000            0.00000000
## Q113583.fctrTalk                          0.00000000            0.00000000
## Q113583.fctrTunes                         0.00000000            0.00000000
## Q113584.fctrPeople                        0.00000000            0.00000000
## Q113584.fctrTechnology                    0.00000000            0.00000000
## Q113992.fctrNo                            0.00000000            0.00000000
## Q113992.fctrYes                           0.00000000            0.00000000
## Q114152.fctrNo                            0.00000000            0.00000000
## Q114152.fctrYes                           0.00000000            0.00000000
## Q114386.fctrMysterious                    0.00000000            0.00000000
## Q114386.fctrTMI                           0.00000000            0.00000000
## Q114517.fctrNo                            0.00000000            0.00000000
## Q114517.fctrYes                           0.00000000            0.00000000
## Q114748.fctrNo                            0.00000000            0.00000000
## Q114748.fctrYes                           0.00000000            0.00000000
## Q114961.fctrNo                            0.00000000            0.00000000
## Q114961.fctrYes                           0.00000000            0.00000000
## Q115195.fctrNo                            0.00000000            0.00000000
## Q115195.fctrYes                           0.00000000            0.00000000
## Q115390.fctrNo                            0.00000000            0.00000000
## Q115602.fctrNo                            0.00000000            0.00000000
## Q115602.fctrYes                           0.00000000            0.00000000
## Q115610.fctrNo                            0.00000000            0.00000000
## Q115610.fctrYes                           0.00000000            0.00000000
## Q115777.fctrEnd                           0.00000000            0.00000000
## Q115777.fctrStart                         0.00000000            0.00000000
## Q115899.fctrMe                            0.00000000            0.00000000
## Q116197.fctrA.M.                          0.00000000            0.00000000
## Q116197.fctrP.M.                          0.00000000            0.00000000
## Q116441.fctrNo                            0.00000000            0.00000000
## Q116441.fctrYes                           0.00000000            0.00000000
## Q116448.fctrNo                            0.00000000            0.00000000
## Q116448.fctrYes                           0.00000000            0.00000000
## Q116601.fctrNo                            0.00000000            0.00000000
## Q116601.fctrYes                           0.00000000            0.00000000
## Q116797.fctrNo                            0.00000000            0.00000000
## Q116797.fctrYes                           0.00000000            0.00000000
## Q116881.fctrHappy                         1.26997411            0.00000000
## Q116953.fctrYes                           0.00000000            0.00000000
## Q117186.fctrCool headed                   0.00000000            0.00000000
## Q117186.fctrHot headed                    0.00000000            0.00000000
## Q117193.fctrOdd hours                     0.00000000            0.00000000
## Q117193.fctrStandard hours                0.00000000            0.00000000
## Q118117.fctrNo                            0.00000000            0.00000000
## Q118117.fctrYes                           0.00000000            0.00000000
## Q118232.fctrPr                            0.00000000            0.00000000
## Q118233.fctrNo                            0.00000000            0.00000000
## Q118233.fctrYes                           0.00000000            0.00000000
## Q118237.fctrNo                            0.00000000            0.00000000
## Q118237.fctrYes                           0.00000000            0.00000000
## Q118892.fctrNo                            0.00000000            0.00000000
## Q118892.fctrYes                           0.00000000            0.00000000
## Q119334.fctrNo                            0.00000000            0.00000000
## Q119334.fctrYes                           0.00000000            0.00000000
## Q119650.fctrGiving                        0.00000000            0.00000000
## Q119650.fctrReceiving                     0.00000000            0.00000000
## Q119851.fctrYes                           0.00000000            0.00000000
## Q120012.fctrNo                            0.00000000            0.00000000
## Q120012.fctrYes                           0.00000000            0.00000000
## Q120014.fctrNo                            0.00000000            0.00000000
## Q120014.fctrYes                           0.00000000            0.00000000
## Q120194.fctrTry first                     0.00000000            0.00000000
## Q120472.fctrArt                           0.00000000            0.00000000
## Q120650.fctrNo                            0.00000000            0.00000000
## Q120650.fctrYes                           0.00000000            0.00000000
## Q120978.fctrNo                            0.00000000            0.00000000
## Q120978.fctrYes                           0.00000000            0.00000000
## Q121011.fctrNo                            0.00000000            0.00000000
## Q121011.fctrYes                           0.00000000            0.00000000
## Q121699.fctrNo                            0.17720767            0.00000000
## Q121700.fctrNo                            0.00000000            0.00000000
## Q121700.fctrYes                           0.00000000            0.00000000
## Q122120.fctrNo                            0.00000000            0.00000000
## Q122120.fctrYes                           0.00000000            0.00000000
## Q122769.fctrNo                            0.00000000            0.00000000
## Q122769.fctrYes                           0.00000000            0.00000000
## Q122770.fctrNo                            0.00000000            0.00000000
## Q122770.fctrYes                           0.00000000            0.00000000
## Q122771.fctrPc                            0.00000000            0.00000000
## Q123464.fctrNo                            0.00000000            0.00000000
## Q123464.fctrYes                           0.00000000            0.00000000
## Q123621.fctrNo                            0.00000000            0.00000000
## Q123621.fctrYes                           0.00000000            0.00000000
## Q124122.fctrNo                            0.00000000            0.00000000
## Q124122.fctrYes                           0.00000000            0.00000000
## Q124742.fctrNo                            0.00000000            0.00000000
## Q124742.fctrYes                           0.00000000            0.00000000
## Q96024.fctrNo                             0.00000000            0.00000000
## Q96024.fctrYes                            0.00000000            0.00000000
## Q98059.fctrOnly-child                     0.00000000            0.00000000
## Q98059.fctrYes                            0.00000000            0.00000000
## Q98078.fctrNo                             0.00000000            0.00000000
## Q98078.fctrYes                            0.00000000            0.00000000
## Q98578.fctrNo                             0.00000000            0.00000000
## Q98578.fctrYes                            0.00000000            0.00000000
## Q98869.fctrYes                            0.00000000            0.00000000
## Q99581.fctrNo                             0.00000000            0.00000000
## Q99581.fctrYes                            0.00000000            0.00000000
## Q99716.fctrNo                             0.00000000            0.00000000
## Q99716.fctrYes                            0.00000000            0.00000000
## Q99982.fctrCheck!                         0.00000000            0.00000000
## Q99982.fctrNope                           0.00000000            0.00000000
## YOB.Age.fctr.C                            0.00000000            0.00000000
## YOB.Age.fctr.Q                            0.00000000            0.00000000
## YOB.Age.fctr^4                            0.00000000            0.00000000
## YOB.Age.fctr^5                            0.00000000            0.00000000
## YOB.Age.fctr^6                            0.00000000            0.00000000
## YOB.Age.fctr^7                            0.00000000            0.00000000
##                                         imp
## Q109244.fctrYes                100.00000000
## Q109244.fctrNo                  37.85598680
## Hhold.fctrPKn                   37.25545617
## Q115611.fctrYes                 34.20502961
## Q98197.fctrNo                   29.31306049
## Q98869.fctrNo                   17.15145611
## Q116881.fctrRight               17.03627425
## Q113181.fctrYes                 15.97468665
## Q115611.fctrNo                  14.63644538
## Gender.fctrM                    11.17712069
## Q113181.fctrNo                   9.56313129
## Q101163.fctrDad                  9.47197006
## Q119851.fctrNo                   8.61619297
## Q118232.fctrId                   7.76417484
## Q120379.fctrYes                  7.36880104
## Q110740.fctrPC                   7.30992383
## Q120472.fctrScience              6.22468445
## Q100689.fctrYes                  5.63783468
## Hhold.fctrMKy                    5.39418970
## Q106997.fctrGr                   5.05907385
## Q122771.fctrPt                   4.61769175
## Q101163.fctrMom                  4.02148246
## Income.fctr.Q                    4.01971777
## Q115390.fctrYes                  3.83631167
## Income.fctr.C                    3.58434097
## Q99480.fctrNo                    3.12790282
## Q115899.fctrCs                   2.81436544
## Q99480.fctrYes                   2.50509763
## Q106042.fctrNo                   2.00082292
## YOB.Age.fctr.L                   1.96027716
## Q120194.fctrStudy first          1.46404807
## Q121699.fctrYes                  1.06835640
## Q112478.fctrNo                   1.03515874
## Edn.fctr.L                       0.98609275
## Q108855.fctrYes!                 0.69851577
## Q116953.fctrNo                   0.40800584
## YOB.Age.fctr^8                   0.36334019
## Q98197.fctrYes                   0.28772824
## Q104996.fctrNo                   0.24396635
## Q106389.fctrNo                   0.11860662
## Q120379.fctrNo                   0.06011541
## Q110740.fctrMac                  0.02569374
## .rnorm                           0.00000000
## Edn.fctr.C                       0.00000000
## Edn.fctr.Q                       0.00000000
## Edn.fctr^4                       0.00000000
## Edn.fctr^5                       0.00000000
## Edn.fctr^6                       0.00000000
## Edn.fctr^7                       0.00000000
## Gender.fctrF                     0.00000000
## Hhold.fctrMKn                    0.00000000
## Hhold.fctrMKn:.clusterid.fctr2   0.00000000
## Hhold.fctrMKn:.clusterid.fctr3   0.00000000
## Hhold.fctrMKn:.clusterid.fctr4   0.00000000
## Hhold.fctrMKy:.clusterid.fctr2   0.00000000
## Hhold.fctrMKy:.clusterid.fctr3   0.00000000
## Hhold.fctrMKy:.clusterid.fctr4   0.00000000
## Hhold.fctrN:.clusterid.fctr2     0.00000000
## Hhold.fctrN:.clusterid.fctr3     0.00000000
## Hhold.fctrN:.clusterid.fctr4     0.00000000
## Hhold.fctrPKn:.clusterid.fctr2   0.00000000
## Hhold.fctrPKn:.clusterid.fctr3   0.00000000
## Hhold.fctrPKn:.clusterid.fctr4   0.00000000
## Hhold.fctrPKy                    0.00000000
## Hhold.fctrPKy:.clusterid.fctr2   0.00000000
## Hhold.fctrPKy:.clusterid.fctr3   0.00000000
## Hhold.fctrPKy:.clusterid.fctr4   0.00000000
## Hhold.fctrSKn                    0.00000000
## Hhold.fctrSKn:.clusterid.fctr2   0.00000000
## Hhold.fctrSKn:.clusterid.fctr3   0.00000000
## Hhold.fctrSKn:.clusterid.fctr4   0.00000000
## Hhold.fctrSKy                    0.00000000
## Hhold.fctrSKy:.clusterid.fctr2   0.00000000
## Hhold.fctrSKy:.clusterid.fctr3   0.00000000
## Hhold.fctrSKy:.clusterid.fctr4   0.00000000
## Income.fctr.L                    0.00000000
## Income.fctr^4                    0.00000000
## Income.fctr^5                    0.00000000
## Income.fctr^6                    0.00000000
## Q100010.fctrNo                   0.00000000
## Q100010.fctrYes                  0.00000000
## Q100562.fctrNo                   0.00000000
## Q100562.fctrYes                  0.00000000
## Q100680.fctrNo                   0.00000000
## Q100680.fctrYes                  0.00000000
## Q100689.fctrNo                   0.00000000
## Q101162.fctrOptimist             0.00000000
## Q101162.fctrPessimist            0.00000000
## Q101596.fctrNo                   0.00000000
## Q101596.fctrYes                  0.00000000
## Q102089.fctrOwn                  0.00000000
## Q102089.fctrRent                 0.00000000
## Q102289.fctrNo                   0.00000000
## Q102289.fctrYes                  0.00000000
## Q102674.fctrNo                   0.00000000
## Q102674.fctrYes                  0.00000000
## Q102687.fctrNo                   0.00000000
## Q102687.fctrYes                  0.00000000
## Q102906.fctrNo                   0.00000000
## Q102906.fctrYes                  0.00000000
## Q103293.fctrNo                   0.00000000
## Q103293.fctrYes                  0.00000000
## Q104996.fctrYes                  0.00000000
## Q105655.fctrNo                   0.00000000
## Q105655.fctrYes                  0.00000000
## Q105840.fctrNo                   0.00000000
## Q105840.fctrYes                  0.00000000
## Q106042.fctrYes                  0.00000000
## Q106272.fctrNo                   0.00000000
## Q106272.fctrYes                  0.00000000
## Q106388.fctrNo                   0.00000000
## Q106388.fctrYes                  0.00000000
## Q106389.fctrYes                  0.00000000
## Q106993.fctrNo                   0.00000000
## Q106993.fctrYes                  0.00000000
## Q106997.fctrYy                   0.00000000
## Q107491.fctrNo                   0.00000000
## Q107491.fctrYes                  0.00000000
## Q107869.fctrNo                   0.00000000
## Q107869.fctrYes                  0.00000000
## Q108342.fctrIn-person            0.00000000
## Q108342.fctrOnline               0.00000000
## Q108343.fctrNo                   0.00000000
## Q108343.fctrYes                  0.00000000
## Q108617.fctrNo                   0.00000000
## Q108617.fctrYes                  0.00000000
## Q108754.fctrNo                   0.00000000
## Q108754.fctrYes                  0.00000000
## Q108855.fctrUmm...               0.00000000
## Q108856.fctrSocialize            0.00000000
## Q108856.fctrSpace                0.00000000
## Q108950.fctrCautious             0.00000000
## Q108950.fctrRisk-friendly        0.00000000
## Q109367.fctrNo                   0.00000000
## Q109367.fctrYes                  0.00000000
## Q111220.fctrNo                   0.00000000
## Q111220.fctrYes                  0.00000000
## Q111580.fctrDemanding            0.00000000
## Q111580.fctrSupportive           0.00000000
## Q111848.fctrNo                   0.00000000
## Q111848.fctrYes                  0.00000000
## Q112270.fctrNo                   0.00000000
## Q112270.fctrYes                  0.00000000
## Q112478.fctrYes                  0.00000000
## Q112512.fctrNo                   0.00000000
## Q112512.fctrYes                  0.00000000
## Q113583.fctrTalk                 0.00000000
## Q113583.fctrTunes                0.00000000
## Q113584.fctrPeople               0.00000000
## Q113584.fctrTechnology           0.00000000
## Q113992.fctrNo                   0.00000000
## Q113992.fctrYes                  0.00000000
## Q114152.fctrNo                   0.00000000
## Q114152.fctrYes                  0.00000000
## Q114386.fctrMysterious           0.00000000
## Q114386.fctrTMI                  0.00000000
## Q114517.fctrNo                   0.00000000
## Q114517.fctrYes                  0.00000000
## Q114748.fctrNo                   0.00000000
## Q114748.fctrYes                  0.00000000
## Q114961.fctrNo                   0.00000000
## Q114961.fctrYes                  0.00000000
## Q115195.fctrNo                   0.00000000
## Q115195.fctrYes                  0.00000000
## Q115390.fctrNo                   0.00000000
## Q115602.fctrNo                   0.00000000
## Q115602.fctrYes                  0.00000000
## Q115610.fctrNo                   0.00000000
## Q115610.fctrYes                  0.00000000
## Q115777.fctrEnd                  0.00000000
## Q115777.fctrStart                0.00000000
## Q115899.fctrMe                   0.00000000
## Q116197.fctrA.M.                 0.00000000
## Q116197.fctrP.M.                 0.00000000
## Q116441.fctrNo                   0.00000000
## Q116441.fctrYes                  0.00000000
## Q116448.fctrNo                   0.00000000
## Q116448.fctrYes                  0.00000000
## Q116601.fctrNo                   0.00000000
## Q116601.fctrYes                  0.00000000
## Q116797.fctrNo                   0.00000000
## Q116797.fctrYes                  0.00000000
## Q116881.fctrHappy                0.00000000
## Q116953.fctrYes                  0.00000000
## Q117186.fctrCool headed          0.00000000
## Q117186.fctrHot headed           0.00000000
## Q117193.fctrOdd hours            0.00000000
## Q117193.fctrStandard hours       0.00000000
## Q118117.fctrNo                   0.00000000
## Q118117.fctrYes                  0.00000000
## Q118232.fctrPr                   0.00000000
## Q118233.fctrNo                   0.00000000
## Q118233.fctrYes                  0.00000000
## Q118237.fctrNo                   0.00000000
## Q118237.fctrYes                  0.00000000
## Q118892.fctrNo                   0.00000000
## Q118892.fctrYes                  0.00000000
## Q119334.fctrNo                   0.00000000
## Q119334.fctrYes                  0.00000000
## Q119650.fctrGiving               0.00000000
## Q119650.fctrReceiving            0.00000000
## Q119851.fctrYes                  0.00000000
## Q120012.fctrNo                   0.00000000
## Q120012.fctrYes                  0.00000000
## Q120014.fctrNo                   0.00000000
## Q120014.fctrYes                  0.00000000
## Q120194.fctrTry first            0.00000000
## Q120472.fctrArt                  0.00000000
## Q120650.fctrNo                   0.00000000
## Q120650.fctrYes                  0.00000000
## Q120978.fctrNo                   0.00000000
## Q120978.fctrYes                  0.00000000
## Q121011.fctrNo                   0.00000000
## Q121011.fctrYes                  0.00000000
## Q121699.fctrNo                   0.00000000
## Q121700.fctrNo                   0.00000000
## Q121700.fctrYes                  0.00000000
## Q122120.fctrNo                   0.00000000
## Q122120.fctrYes                  0.00000000
## Q122769.fctrNo                   0.00000000
## Q122769.fctrYes                  0.00000000
## Q122770.fctrNo                   0.00000000
## Q122770.fctrYes                  0.00000000
## Q122771.fctrPc                   0.00000000
## Q123464.fctrNo                   0.00000000
## Q123464.fctrYes                  0.00000000
## Q123621.fctrNo                   0.00000000
## Q123621.fctrYes                  0.00000000
## Q124122.fctrNo                   0.00000000
## Q124122.fctrYes                  0.00000000
## Q124742.fctrNo                   0.00000000
## Q124742.fctrYes                  0.00000000
## Q96024.fctrNo                    0.00000000
## Q96024.fctrYes                   0.00000000
## Q98059.fctrOnly-child            0.00000000
## Q98059.fctrYes                   0.00000000
## Q98078.fctrNo                    0.00000000
## Q98078.fctrYes                   0.00000000
## Q98578.fctrNo                    0.00000000
## Q98578.fctrYes                   0.00000000
## Q98869.fctrYes                   0.00000000
## Q99581.fctrNo                    0.00000000
## Q99581.fctrYes                   0.00000000
## Q99716.fctrNo                    0.00000000
## Q99716.fctrYes                   0.00000000
## Q99982.fctrCheck!                0.00000000
## Q99982.fctrNope                  0.00000000
## YOB.Age.fctr.C                   0.00000000
## YOB.Age.fctr.Q                   0.00000000
## YOB.Age.fctr^4                   0.00000000
## YOB.Age.fctr^5                   0.00000000
## YOB.Age.fctr^6                   0.00000000
## YOB.Age.fctr^7                   0.00000000

if (glb_is_classification && glb_is_binomial)
    glb_analytics_diag_plots(obs_df=glbObsTrn, mdl_id=glbMdlFinId, 
            prob_threshold=glb_models_df[glb_models_df$id == glbMdlSelId, 
                                         "opt.prob.threshold.OOB"]) else
    glb_analytics_diag_plots(obs_df=glbObsTrn, mdl_id=glbMdlFinId)

## Warning in glb_analytics_diag_plots(obs_df = glbObsTrn, mdl_id =
## glbMdlFinId, : Limiting important feature scatter plots to 5 out of 108

## [1] "Min/Max Boundaries: "
## [1] "Inaccurate: "
##   USER_ID Party.fctr Party.fctr.All.X..rcv.glmnet.prob
## 1    1309          D                         0.3020170
## 2    4956          D                         0.3019760
## 3    3578          D                         0.3017465
## 4    1393          D                                NA
## 5    1311          D                         0.3051844
## 6    3006          D                         0.3385111
##   Party.fctr.All.X..rcv.glmnet Party.fctr.All.X..rcv.glmnet.err
## 1                            R                             TRUE
## 2                            R                             TRUE
## 3                            R                             TRUE
## 4                         <NA>                               NA
## 5                            R                             TRUE
## 6                            R                             TRUE
##   Party.fctr.All.X..rcv.glmnet.err.abs Party.fctr.All.X..rcv.glmnet.is.acc
## 1                            0.6979830                               FALSE
## 2                            0.6980240                               FALSE
## 3                            0.6982535                               FALSE
## 4                                   NA                                  NA
## 5                            0.6948156                               FALSE
## 6                            0.6614889                               FALSE
##   Party.fctr.Final..rcv.glmnet.prob Party.fctr.Final..rcv.glmnet
## 1                         0.2299076                            R
## 2                         0.2484385                            R
## 3                         0.2492683                            R
## 4                         0.2514118                            R
## 5                         0.2537294                            R
## 6                         0.2550309                            R
##   Party.fctr.Final..rcv.glmnet.err Party.fctr.Final..rcv.glmnet.err.abs
## 1                             TRUE                            0.7700924
## 2                             TRUE                            0.7515615
## 3                             TRUE                            0.7507317
## 4                             TRUE                            0.7485882
## 5                             TRUE                            0.7462706
## 6                             TRUE                            0.7449691
##   Party.fctr.Final..rcv.glmnet.is.acc
## 1                               FALSE
## 2                               FALSE
## 3                               FALSE
## 4                               FALSE
## 5                               FALSE
## 6                               FALSE
##   Party.fctr.Final..rcv.glmnet.accurate Party.fctr.Final..rcv.glmnet.error
## 1                                 FALSE                         -0.4700924
## 2                                 FALSE                         -0.4515615
## 3                                 FALSE                         -0.4507317
## 4                                 FALSE                         -0.4485882
## 5                                 FALSE                         -0.4462706
## 6                                 FALSE                         -0.4449691
##      USER_ID Party.fctr Party.fctr.All.X..rcv.glmnet.prob
## 804     1249          D                         0.4698030
## 1053    2071          D                                NA
## 1436    3980          D                         0.5439720
## 1709    1369          D                         0.5353958
## 1980    5219          D                         0.5827582
## 2282    4762          R                         0.6827667
##      Party.fctr.All.X..rcv.glmnet Party.fctr.All.X..rcv.glmnet.err
## 804                             R                             TRUE
## 1053                         <NA>                               NA
## 1436                            R                             TRUE
## 1709                            R                             TRUE
## 1980                            R                             TRUE
## 2282                            R                            FALSE
##      Party.fctr.All.X..rcv.glmnet.err.abs
## 804                             0.5301970
## 1053                                   NA
## 1436                            0.4560280
## 1709                            0.4646042
## 1980                            0.4172418
## 2282                            0.6827667
##      Party.fctr.All.X..rcv.glmnet.is.acc Party.fctr.Final..rcv.glmnet.prob
## 804                                FALSE                         0.4975030
## 1053                                  NA                         0.5184046
## 1436                               FALSE                         0.5439207
## 1709                               FALSE                         0.5647558
## 1980                               FALSE                         0.6033227
## 2282                                TRUE                         0.7245643
##      Party.fctr.Final..rcv.glmnet Party.fctr.Final..rcv.glmnet.err
## 804                             R                             TRUE
## 1053                            R                             TRUE
## 1436                            R                             TRUE
## 1709                            R                             TRUE
## 1980                            R                             TRUE
## 2282                            D                             TRUE
##      Party.fctr.Final..rcv.glmnet.err.abs
## 804                             0.5024970
## 1053                            0.4815954
## 1436                            0.4560793
## 1709                            0.4352442
## 1980                            0.3966773
## 2282                            0.7245643
##      Party.fctr.Final..rcv.glmnet.is.acc
## 804                                FALSE
## 1053                               FALSE
## 1436                               FALSE
## 1709                               FALSE
## 1980                               FALSE
## 2282                               FALSE
##      Party.fctr.Final..rcv.glmnet.accurate
## 804                                  FALSE
## 1053                                 FALSE
## 1436                                 FALSE
## 1709                                 FALSE
## 1980                                 FALSE
## 2282                                 FALSE
##      Party.fctr.Final..rcv.glmnet.error
## 804                         -0.20249705
## 1053                        -0.18159544
## 1436                        -0.15607928
## 1709                        -0.13524417
## 1980                        -0.09667733
## 2282                         0.02456433
##      USER_ID Party.fctr Party.fctr.All.X..rcv.glmnet.prob
## 2413    1307          R                                NA
## 2414    1236          R                         0.8147302
## 2415    1515          R                         0.8208858
## 2416    2749          R                         0.8214803
## 2417     626          R                         0.8149343
## 2418    3895          R                         0.8341916
##      Party.fctr.All.X..rcv.glmnet Party.fctr.All.X..rcv.glmnet.err
## 2413                         <NA>                               NA
## 2414                            D                             TRUE
## 2415                            D                             TRUE
## 2416                            D                             TRUE
## 2417                            D                             TRUE
## 2418                            D                             TRUE
##      Party.fctr.All.X..rcv.glmnet.err.abs
## 2413                                   NA
## 2414                            0.8147302
## 2415                            0.8208858
## 2416                            0.8214803
## 2417                            0.8149343
## 2418                            0.8341916
##      Party.fctr.All.X..rcv.glmnet.is.acc Party.fctr.Final..rcv.glmnet.prob
## 2413                                  NA                         0.8587457
## 2414                               FALSE                         0.8636172
## 2415                               FALSE                         0.8675383
## 2416                               FALSE                         0.8707850
## 2417                               FALSE                         0.8762783
## 2418                               FALSE                         0.8843825
##      Party.fctr.Final..rcv.glmnet Party.fctr.Final..rcv.glmnet.err
## 2413                            D                             TRUE
## 2414                            D                             TRUE
## 2415                            D                             TRUE
## 2416                            D                             TRUE
## 2417                            D                             TRUE
## 2418                            D                             TRUE
##      Party.fctr.Final..rcv.glmnet.err.abs
## 2413                            0.8587457
## 2414                            0.8636172
## 2415                            0.8675383
## 2416                            0.8707850
## 2417                            0.8762783
## 2418                            0.8843825
##      Party.fctr.Final..rcv.glmnet.is.acc
## 2413                               FALSE
## 2414                               FALSE
## 2415                               FALSE
## 2416                               FALSE
## 2417                               FALSE
## 2418                               FALSE
##      Party.fctr.Final..rcv.glmnet.accurate
## 2413                                 FALSE
## 2414                                 FALSE
## 2415                                 FALSE
## 2416                                 FALSE
## 2417                                 FALSE
## 2418                                 FALSE
##      Party.fctr.Final..rcv.glmnet.error
## 2413                          0.1587457
## 2414                          0.1636172
## 2415                          0.1675383
## 2416                          0.1707850
## 2417                          0.1762783
## 2418                          0.1843825

dsp_feats_vctr <- c(NULL)
for(var in grep(".imp", names(glb_feats_df), fixed=TRUE, value=TRUE))
    dsp_feats_vctr <- union(dsp_feats_vctr, 
                            glb_feats_df[!is.na(glb_feats_df[, var]), "id"])

# print(glbObsTrn[glbObsTrn$UniqueID %in% FN_OOB_ids, 
#                     grep(glb_rsp_var, names(glbObsTrn), value=TRUE)])

print(setdiff(names(glbObsTrn), names(glbObsAll)))

## [1] "Party.fctr.Final..rcv.glmnet.prob"   
## [2] "Party.fctr.Final..rcv.glmnet"        
## [3] "Party.fctr.Final..rcv.glmnet.err"    
## [4] "Party.fctr.Final..rcv.glmnet.err.abs"
## [5] "Party.fctr.Final..rcv.glmnet.is.acc"

for (col in setdiff(names(glbObsTrn), names(glbObsAll)))
    # Merge or cbind ?
    glbObsAll[glbObsAll$.src == "Train", col] <- glbObsTrn[, col]

print(setdiff(names(glbObsFit), names(glbObsAll)))

## character(0)

print(setdiff(names(glbObsOOB), names(glbObsAll)))

## character(0)

for (col in setdiff(names(glbObsOOB), names(glbObsAll)))
    # Merge or cbind ?
    glbObsAll[glbObsAll$.lcn == "OOB", col] <- glbObsOOB[, col]
    
print(setdiff(names(glbObsNew), names(glbObsAll)))

## character(0)

#glb2Sav(); all.equal(savObsAll, glbObsAll); all.equal(sav_models_lst, glb_models_lst)
#load(file = paste0(glbOut$pfx, "dsk_knitr.RData"))
#cmpCols <- names(glbObsAll)[!grepl("\\.Final\\.", names(glbObsAll))]; all.equal(savObsAll[, cmpCols], glbObsAll[, cmpCols]); all.equal(savObsAll[, "H.P.http"], glbObsAll[, "H.P.http"]); 

replay.petrisim(pn = glb_analytics_pn, 
    replay.trans = (glb_analytics_avl_objs <- c(glb_analytics_avl_objs, 
        "data.training.all.prediction","model.final")), flip_coord = TRUE)

## time trans    "bgn " "fit.data.training.all " "predict.data.new " "end " 
## 0.0000   multiple enabled transitions:  data.training.all data.new model.selected    firing:  model.selected 
## 1.0000    3   2 1 0 0 
## 1.0000   multiple enabled transitions:  data.training.all data.new model.selected model.final data.training.all.prediction   firing:  data.training.all.prediction 
## 2.0000    5   2 0 0 1

## Warning in replay.petrisim(pn = glb_analytics_pn, replay.trans =
## (glb_analytics_avl_objs <- c(glb_analytics_avl_objs, : Transition:
## model.final not enabled; adding missing token(s)

## Warning in replay.petrisim(pn = glb_analytics_pn, replay.trans
## = (glb_analytics_avl_objs <- c(glb_analytics_avl_objs, : Place:
## fit.data.training.all: added 1 missing token

## 2.0000   multiple enabled transitions:  data.training.all data.new model.selected model.final data.training.all.prediction   firing:  model.final 
## 3.0000    4   2 0 1 1

glb_chunks_df <- myadd_chunk(glb_chunks_df, "predict.data.new", major.inc = TRUE)

##                label step_major step_minor label_minor     bgn     end
## 9  fit.data.training          5          1           1 470.986 481.815
## 10  predict.data.new          6          0           0 481.816      NA
##    elapsed
## 9    10.83
## 10      NA

Step `6.0: predict data new`

## Warning in glb_get_predictions(obs_df, mdl_id = glbMdlFinId, rsp_var =
## glb_rsp_var, : Using default probability threshold: 0.7

## Warning in glb_get_predictions(obs_df, mdl_id = glbMdlFinId, rsp_var =
## glb_rsp_var, : Using default probability threshold: 0.7

## Warning in glb_analytics_diag_plots(obs_df = glbObsNew, mdl_id =
## glbMdlFinId, : Limiting important feature scatter plots to 5 out of 108

## Warning: Removed 1392 rows containing missing values (geom_point).

## Warning: Removed 1392 rows containing missing values (geom_point).

## Warning: Removed 1392 rows containing missing values (geom_point).

## Warning: Removed 1392 rows containing missing values (geom_point).

## Warning: Removed 1392 rows containing missing values (geom_point).

## Warning: Removed 1392 rows containing missing values (geom_point).

## Warning: Removed 1392 rows containing missing values (geom_point).

## Warning: Removed 1392 rows containing missing values (geom_point).

## Warning: Removed 1392 rows containing missing values (geom_point).

## Warning: Removed 1392 rows containing missing values (geom_point).

## NULL

## [1] "OOBobs total range outliers: 0"
## [1] "newobs total range outliers: 0"

## [1] 0.7

## [1] "glbMdlSelId: All.X##rcv#glmnet"

## [1] "glbMdlFinId: Final##rcv#glmnet"

## [1] "Cross Validation issues:"
##        MFO###myMFO_classfr  Random###myrandom_classfr 
##                          0                          0 
## Max.cor.Y.rcv.1X1###glmnet 
##                          0

##                                 max.Accuracy.OOB max.AUCROCR.OOB
## Interact.High.cor.Y##rcv#glmnet        0.5732143       0.3571392
## All.X##rcv#glm                         0.5723214       0.3358234
## Max.cor.Y##rcv#rpart                   0.5633929       0.3774772
## Max.cor.Y.rcv.1X1###glmnet             0.5633929       0.3658672
## Low.cor.X##rcv#glmnet                  0.5544643       0.3184875
## All.X##rcv#glmnet                      0.5544643       0.3184875
## Random###myrandom_classfr              0.4696429       0.5191202
## MFO###myMFO_classfr                    0.4696429       0.5000000
## Final##rcv#glmnet                             NA              NA
##                                 max.AUCpROC.OOB max.Accuracy.fit
## Interact.High.cor.Y##rcv#glmnet       0.6031353        0.6058167
## All.X##rcv#glm                        0.5947690        0.6029697
## Max.cor.Y##rcv#rpart                  0.5896897        0.6000450
## Max.cor.Y.rcv.1X1###glmnet            0.5896897        0.5721673
## Low.cor.X##rcv#glmnet                 0.6237374        0.6241754
## All.X##rcv#glmnet                     0.6237374        0.6241754
## Random###myrandom_classfr             0.5235690        0.4700989
## MFO###myMFO_classfr                   0.5000000        0.4700989
## Final##rcv#glmnet                            NA        0.6329006
##                                 opt.prob.threshold.fit
## Interact.High.cor.Y##rcv#glmnet                   0.65
## All.X##rcv#glm                                    0.65
## Max.cor.Y##rcv#rpart                              0.55
## Max.cor.Y.rcv.1X1###glmnet                        0.60
## Low.cor.X##rcv#glmnet                             0.60
## All.X##rcv#glmnet                                 0.60
## Random###myrandom_classfr                         0.55
## MFO###myMFO_classfr                               0.50
## Final##rcv#glmnet                                 0.60
##                                 opt.prob.threshold.OOB
## Interact.High.cor.Y##rcv#glmnet                   0.65
## All.X##rcv#glm                                    0.75
## Max.cor.Y##rcv#rpart                              0.55
## Max.cor.Y.rcv.1X1###glmnet                        0.60
## Low.cor.X##rcv#glmnet                             0.70
## All.X##rcv#glmnet                                 0.70
## Random###myrandom_classfr                         0.55
## MFO###myMFO_classfr                               0.50
## Final##rcv#glmnet                                   NA

## [1] "All.X##rcv#glmnet OOB confusion matrix & accuracy: "
##          Prediction
## Reference   R   D
##         R 494  32
##         D 467 127

##     err.abs.fit.sum err.abs.OOB.sum err.abs.trn.sum err.abs.new.sum
## PKy        24.71367         4.28726        28.94117              NA
## PKn        59.07825        14.10653        71.45776              NA
## N         173.80353        38.82857       209.29570              NA
## SKn       884.82425       238.72729      1101.68471              NA
## SKy        65.43207        24.55390        87.66767              NA
## MKn       235.21548        62.21564       292.01933              NA
## MKy       595.03188       134.54661       712.15620              NA
##     .freqRatio.Fit .freqRatio.OOB .freqRatio.Tst .n.Fit .n.New.D .n.New.R
## PKy     0.01169065    0.008035714    0.007183908     52        2        8
## PKn     0.03372302    0.026785714    0.026580460    150       11       26
## N       0.08250899    0.074107143    0.073275862    367        9       93
## SKn     0.43165468    0.456250000    0.458333333   1920      107      531
## SKy     0.03304856    0.047321429    0.046695402    147        9       56
## MKn     0.11600719    0.121428571    0.121408046    516       24      145
## MKy     0.29136691    0.266071429    0.266522989   1296       41      330
##     .n.OOB .n.Trn.D .n.Trn.R .n.Tst .n.fit .n.new .n.trn err.abs.OOB.mean
## PKy      9       35       26     10     52     10     61        0.4763622
## PKn     30      131       49     37    150     37    180        0.4702178
## N       83      230      220    102    367    102    450        0.4678140
## SKn    511     1340     1091    638   1920    638   2431        0.4671767
## SKy     53      119       81     65    147     65    200        0.4632812
## MKn    136      344      308    169    516    169    652        0.4574680
## MKy    298      752      842    371   1296    371   1594        0.4514987
##     err.abs.fit.mean err.abs.new.mean err.abs.trn.mean
## PKy        0.4752630               NA        0.4744454
## PKn        0.3938550               NA        0.3969876
## N          0.4735791               NA        0.4651016
## SKn        0.4608460               NA        0.4531817
## SKy        0.4451161               NA        0.4383384
## MKn        0.4558439               NA        0.4478824
## MKy        0.4591295               NA        0.4467730
##  err.abs.fit.sum  err.abs.OOB.sum  err.abs.trn.sum  err.abs.new.sum 
##      2038.099138       517.265807      2503.222539               NA 
##   .freqRatio.Fit   .freqRatio.OOB   .freqRatio.Tst           .n.Fit 
##         1.000000         1.000000         1.000000      4448.000000 
##         .n.New.D         .n.New.R           .n.OOB         .n.Trn.D 
##       203.000000      1189.000000      1120.000000      2951.000000 
##         .n.Trn.R           .n.Tst           .n.fit           .n.new 
##      2617.000000      1392.000000      4448.000000      1392.000000 
##           .n.trn err.abs.OOB.mean err.abs.fit.mean err.abs.new.mean 
##      5568.000000         3.253819         3.163633               NA 
## err.abs.trn.mean 
##         3.122710

## [1] "Features Importance for selected models:"

##                   All.X..rcv.glmnet.imp Final..rcv.glmnet.imp
## Q109244.fctrYes              100.000000             100.00000
## Hhold.fctrPKn                 41.566427              37.25546
## Q109244.fctrNo                39.809336              37.85599
## Q115611.fctrYes               38.973736              34.20503
## Q98197.fctrNo                 20.633416              29.31306
## Q98869.fctrNo                 20.364904              17.15146
## Q116881.fctrRight             10.343396              17.03627
## Q113181.fctrYes               10.032156              15.97469
## Q115611.fctrNo                 9.696597              14.63645
## Gender.fctrM                   7.316719              11.17712

## [1] "glbObsNew prediction stats:"

## 
##    R    D 
## 1189  203

##                   label step_major step_minor label_minor     bgn    end
## 10     predict.data.new          6          0           0 481.816 500.46
## 11 display.session.info          7          0           0 500.461     NA
##    elapsed
## 10  18.644
## 11      NA

Null Hypothesis ($\sf{H_{0}}$): mpg is not impacted by am_fctr.
The variance by am_fctr appears to be independent. #{r q1, cache=FALSE} # print(t.test(subset(cars_df, am_fctr == "automatic")$mpg, # subset(cars_df, am_fctr == "manual")$mpg, # var.equal=FALSE)$conf) # We reject the null hypothesis i.e. we have evidence to conclude that am_fctr impacts mpg (95% confidence). Manual transmission is better for miles per gallon versus automatic transmission.

##                      label step_major step_minor label_minor     bgn
## 1             cluster.data          1          0           0   9.146
## 2  partition.data.training          2          0           0 137.717
## 4               fit.models          4          0           0 259.549
## 5               fit.models          4          1           1 341.558
## 8        fit.data.training          5          0           0 433.741
## 10        predict.data.new          6          0           0 481.816
## 6               fit.models          4          2           2 415.955
## 9        fit.data.training          5          1           1 470.986
## 3          select.features          3          0           0 253.568
## 7               fit.models          4          3           3 429.047
##        end elapsed duration
## 1  137.716 128.570  128.570
## 2  253.567 115.850  115.850
## 4  341.558  82.009   82.009
## 5  415.955  74.397   74.397
## 8  470.985  37.244   37.244
## 10 500.460  18.644   18.644
## 6  429.046  13.092   13.091
## 9  481.815  10.830   10.829
## 3  259.548   5.981    5.980
## 7  433.740   4.693    4.693
## [1] "Total Elapsed Time: 500.46 secs"

Show of Hands:Votes: Party classification:: Votes_Q_02_cluster

bdanalytics

Introduction:

Synopsis:

Prediction Accuracy Enhancement Options:

[](.png)

Potential next steps include:

Analysis:

Step 1.0: cluster data

chunk option: eval=

Step 1.0: cluster data

Step 1.0: cluster data

Step 1.0: cluster data

Step 1.0: cluster data

Step 1.0: cluster data

Step 1.0: cluster data

```{r extract.features.image, cache=FALSE, echo=FALSE, fig.height=5, fig.width=5, eval=myevlChunk(glbChunks, glbOut$pfx)}

Step 1.0: cluster data

Step 1.0: cluster data

Step 1.0: cluster data

Step 1.0: cluster data

Step 1.0: cluster data

Step 1.0: cluster data

{r cluster.data, cache=FALSE, echo=FALSE, eval=myevlChunk(glbChunks, glbOut$pfx)} #{r cluster.data, cache=FALSE, echo=FALSE, eval=myevlChunk(glbChunks, glbOut$pfx, keep = c(“glbFeatsCategory”,“glb_dsp_cols”))}

Step 2.0: partition data training

Step 3.0: select features

Step 4.0: fit models

Step 5.0: fit data training

Step 6.0: predict data new

Step `1.0: cluster data`

Step `1.0: cluster data`

Step `1.0: cluster data`

Step `1.0: cluster data`

Step `1.0: cluster data`

Step `1.0: cluster data`

Step `1.0: cluster data`

Step `1.0: cluster data`

Step `1.0: cluster data`

Step `1.0: cluster data`

Step `1.0: cluster data`

Step `1.0: cluster data`

Step `1.0: cluster data`

`{r cluster.data, cache=FALSE, echo=FALSE, eval=myevlChunk(glbChunks, glbOut$pfx)} #`{r cluster.data, cache=FALSE, echo=FALSE, eval=myevlChunk(glbChunks, glbOut$pfx, keep = c(“glbFeatsCategory”,“glb_dsp_cols”))}

Step `2.0: partition data training`

Step `3.0: select features`

Step `4.0: fit models`

Step `5.0: fit data training`

Step `6.0: predict data new`